2024-01-11T21:33:18.635 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-11T21:33:18.636 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820 --verbose --name yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi --description rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820/orig.config.yaml 2024-01-11T21:33:18.668 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-11T21:33:18.747 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820 branch: quincy description: rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic} email: null first_in_suite: false job_id: '7513820' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: quincy ceph: conf: client: rbd default format: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(CACHE_POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) sha1: d900343f97c65fbc993efbb0bd33f08284a8d78b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d900343f97c65fbc993efbb0bd33f08284a8d78b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-63937-quincy sha1: 49908e15c7044e78475683aa6ccd77bd187be2f4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7078 sha1: d900343f97c65fbc993efbb0bd33f08284a8d78b sleep_before_teardown: 0 subset: 111/12000 suite: rbd suite_branch: wip-63937-quincy suite_path: /home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: 49908e15c7044e78475683aa6ccd77bd187be2f4 targets: smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create cache 4 - sudo ceph osd tier add rbd cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay rbd cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - workunit: clients: client.0: - rbd/cli_generic.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-11_21:24:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-11T21:33:18.747 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa; will attempt to use it 2024-01-11T21:33:18.748 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks 2024-01-11T21:33:18.748 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-11T21:33:18.749 INFO:teuthology.task.internal:Checking packages... 2024-01-11T21:33:18.775 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd900343f97c65fbc993efbb0bd33f08284a8d78b' 2024-01-11T21:33:18.775 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T21:33:18.775 INFO:teuthology.packaging:ref: None 2024-01-11T21:33:18.775 INFO:teuthology.packaging:tag: None 2024-01-11T21:33:18.775 INFO:teuthology.packaging:branch: quincy 2024-01-11T21:33:18.775 INFO:teuthology.packaging:sha1: d900343f97c65fbc993efbb0bd33f08284a8d78b 2024-01-11T21:33:18.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=quincy 2024-01-11T21:33:19.079 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-272-g6454a2bc-1focal 2024-01-11T21:33:19.081 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-11T21:33:19.086 INFO:teuthology.task.internal:no buildpackages task found 2024-01-11T21:33:19.087 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-11T21:33:19.154 INFO:teuthology.task.internal:Saving configuration 2024-01-11T21:33:19.168 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-11T21:33:19.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T21:33:19.199 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 21:26:49.077035', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-01-11T21:33:19.199 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-11T21:33:19.235 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-11T21:33:19.235 INFO:teuthology.run_tasks:Running task console_log... 2024-01-11T21:33:19.282 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb819cbe310>, signals=[15]) 2024-01-11T21:33:19.283 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-11T21:33:19.288 INFO:teuthology.task.internal:Opening connections... 2024-01-11T21:33:19.288 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-01-11T21:33:19.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T21:33:19.393 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-11T21:33:19.399 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-01-11T21:33:19.522 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-01-11T21:33:19.522 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-01-11T21:33:19.527 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="Ubuntu" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:ID=ubuntu 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE=debian 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="20.04" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://www.ubuntu.com/" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_CODENAME=focal 2024-01-11T21:33:19.528 INFO:teuthology.orchestra.run.smithi178.stdout:UBUNTU_CODENAME=focal 2024-01-11T21:33:19.529 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-01-11T21:33:19.559 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-11T21:33:19.567 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-11T21:33:19.594 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-11T21:33:19.594 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-01-11T21:33:19.600 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-11T21:33:19.624 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-11T21:33:19.624 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-01-11T21:33:19.637 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-11T21:33:19.678 INFO:teuthology.run_tasks:Running task kernel... 2024-01-11T21:33:19.722 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-11T21:33:19.722 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-11T21:33:19.723 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-11T21:33:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T21:33:19.728 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-01-11T21:33:19.734 INFO:teuthology.orchestra.run.smithi178.stdout:5.4.0-124-generic 2024-01-11T21:33:19.735 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.4.0-124-generic 2024-01-11T21:33:19.735 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2024-01-11T21:33:19.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-01-11T21:33:19.993 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-01-11T21:33:19.993 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2024-01-11T21:33:19.993 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2024-01-11T21:33:20.071 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2024-01-11T21:33:20.405 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,647 kB] 2024-01-11T21:33:20.565 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [920 kB] 2024-01-11T21:33:20.565 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [694 kB] 2024-01-11T21:33:20.575 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [406 kB] 2024-01-11T21:33:20.580 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2024-01-11T21:33:20.580 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,461 kB] 2024-01-11T21:33:20.608 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [35.1 kB] 2024-01-11T21:33:20.608 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [343 kB] 2024-01-11T21:33:20.610 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2024-01-11T21:33:20.611 INFO:teuthology.orchestra.run.smithi178.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [921 kB] 2024-01-11T21:33:20.623 INFO:teuthology.orchestra.run.smithi178.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [637 kB] 2024-01-11T21:33:20.630 INFO:teuthology.orchestra.run.smithi178.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [193 kB] 2024-01-11T21:33:20.633 INFO:teuthology.orchestra.run.smithi178.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-01-11T21:33:20.638 INFO:teuthology.orchestra.run.smithi178.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2024-01-11T21:33:20.639 INFO:teuthology.orchestra.run.smithi178.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.9 kB] 2024-01-11T21:33:20.639 INFO:teuthology.orchestra.run.smithi178.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,796 B] 2024-01-11T21:33:20.639 INFO:teuthology.orchestra.run.smithi178.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2024-01-11T21:33:20.680 INFO:teuthology.orchestra.run.smithi178.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,028 kB] 2024-01-11T21:33:20.744 INFO:teuthology.orchestra.run.smithi178.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [489 kB] 2024-01-11T21:33:20.749 INFO:teuthology.orchestra.run.smithi178.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2024-01-11T21:33:20.750 INFO:teuthology.orchestra.run.smithi178.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,579 kB] 2024-01-11T21:33:20.779 INFO:teuthology.orchestra.run.smithi178.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.4 kB] 2024-01-11T21:33:20.779 INFO:teuthology.orchestra.run.smithi178.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [360 kB] 2024-01-11T21:33:20.783 INFO:teuthology.orchestra.run.smithi178.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2024-01-11T21:33:20.784 INFO:teuthology.orchestra.run.smithi178.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [763 kB] 2024-01-11T21:33:20.793 INFO:teuthology.orchestra.run.smithi178.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,144 kB] 2024-01-11T21:33:20.808 INFO:teuthology.orchestra.run.smithi178.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [274 kB] 2024-01-11T21:33:20.810 INFO:teuthology.orchestra.run.smithi178.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2024-01-11T21:33:20.812 INFO:teuthology.orchestra.run.smithi178.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-01-11T21:33:20.812 INFO:teuthology.orchestra.run.smithi178.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [26.1 kB] 2024-01-11T21:33:20.813 INFO:teuthology.orchestra.run.smithi178.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,768 B] 2024-01-11T21:33:20.813 INFO:teuthology.orchestra.run.smithi178.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2024-01-11T21:33:20.830 INFO:teuthology.orchestra.run.smithi178.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-01-11T21:33:20.831 INFO:teuthology.orchestra.run.smithi178.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-01-11T21:33:20.832 INFO:teuthology.orchestra.run.smithi178.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-01-11T21:33:20.832 INFO:teuthology.orchestra.run.smithi178.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-01-11T21:33:20.832 INFO:teuthology.orchestra.run.smithi178.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-01-11T21:33:20.833 INFO:teuthology.orchestra.run.smithi178.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-01-11T21:33:24.612 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 18.5 MB in 3s (6,884 kB/s) 2024-01-11T21:33:26.111 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-01-11T21:33:26.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-01-11T21:33:26.224 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-01-11T21:33:26.428 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-01-11T21:33:26.429 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-01-11T21:33:26.621 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2024-01-11T21:33:26.621 INFO:teuthology.orchestra.run.smithi178.stdout: linux-generic linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2024-01-11T21:33:26.622 INFO:teuthology.orchestra.run.smithi178.stdout: linux-headers-generic linux-image-5.4.0-169-generic 2024-01-11T21:33:26.622 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-5.4.0-169-generic linux-modules-extra-5.4.0-169-generic 2024-01-11T21:33:26.623 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2024-01-11T21:33:26.623 INFO:teuthology.orchestra.run.smithi178.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-01-11T21:33:26.623 INFO:teuthology.orchestra.run.smithi178.stdout:Recommended packages: 2024-01-11T21:33:26.623 INFO:teuthology.orchestra.run.smithi178.stdout: thermald 2024-01-11T21:33:26.676 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2024-01-11T21:33:26.676 INFO:teuthology.orchestra.run.smithi178.stdout: linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2024-01-11T21:33:26.676 INFO:teuthology.orchestra.run.smithi178.stdout: linux-image-5.4.0-169-generic linux-modules-5.4.0-169-generic 2024-01-11T21:33:26.676 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-extra-5.4.0-169-generic 2024-01-11T21:33:26.677 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages will be upgraded: 2024-01-11T21:33:26.678 INFO:teuthology.orchestra.run.smithi178.stdout: linux-generic linux-headers-generic linux-image-generic 2024-01-11T21:33:26.737 INFO:teuthology.orchestra.run.smithi178.stdout:3 upgraded, 5 newly installed, 0 to remove and 282 not upgraded. 2024-01-11T21:33:26.738 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 77.1 MB of archives. 2024-01-11T21:33:26.738 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 380 MB of additional disk space will be used. 2024-01-11T21:33:26.738 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-169-generic amd64 5.4.0-169.187 [15.0 MB] 2024-01-11T21:33:27.139 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-169-generic amd64 5.4.0-169.187 [10.5 MB] 2024-01-11T21:33:27.252 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-169-generic amd64 5.4.0-169.187 [39.2 MB] 2024-01-11T21:33:27.834 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.169.167 [1,900 B] 2024-01-11T21:33:27.834 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.169.167 [2,520 B] 2024-01-11T21:33:27.834 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169 all 5.4.0-169.187 [11.0 MB] 2024-01-11T21:33:28.015 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169-generic amd64 5.4.0-169.187 [1,374 kB] 2024-01-11T21:33:28.031 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.169.167 [2,392 B] 2024-01-11T21:33:28.765 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 77.1 MB in 1s (57.3 MB/s) 2024-01-11T21:33:28.975 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-5.4.0-169-generic. 2024-01-11T21:33:30.717 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-01-11T21:33:30.722 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../0-linux-modules-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2024-01-11T21:33:30.856 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:32.816 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-image-5.4.0-169-generic. 2024-01-11T21:33:32.829 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../1-linux-image-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2024-01-11T21:33:32.921 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:34.011 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-169-generic. 2024-01-11T21:33:34.026 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2024-01-11T21:33:34.069 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:38.842 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../3-linux-generic_5.4.0.169.167_amd64.deb ... 2024-01-11T21:33:38.952 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2024-01-11T21:33:39.318 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.169.167_amd64.deb ... 2024-01-11T21:33:39.472 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2024-01-11T21:33:39.808 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-headers-5.4.0-169. 2024-01-11T21:33:39.822 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../5-linux-headers-5.4.0-169_5.4.0-169.187_all.deb ... 2024-01-11T21:33:39.858 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-5.4.0-169 (5.4.0-169.187) ... 2024-01-11T21:33:45.401 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-headers-5.4.0-169-generic. 2024-01-11T21:33:45.423 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../6-linux-headers-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2024-01-11T21:33:45.468 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:48.696 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.169.167_amd64.deb ... 2024-01-11T21:33:48.823 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2024-01-11T21:33:49.181 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-5.4.0-169 (5.4.0-169.187) ... 2024-01-11T21:33:49.286 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:49.412 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:50.657 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:52.064 INFO:teuthology.orchestra.run.smithi178.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-169-generic 2024-01-11T21:33:52.064 INFO:teuthology.orchestra.run.smithi178.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-169-generic 2024-01-11T21:33:52.376 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-generic (5.4.0.169.167) ... 2024-01-11T21:33:52.502 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:53.653 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-generic (5.4.0.169.167) ... 2024-01-11T21:33:53.813 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-generic (5.4.0.169.167) ... 2024-01-11T21:33:53.940 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2024-01-11T21:33:54.014 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-01-11T21:33:54.014 INFO:teuthology.orchestra.run.smithi178.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-169-generic 2024-01-11T21:34:16.217 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-01-11T21:34:16.217 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub' 2024-01-11T21:34:16.231 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-11T21:34:16.251 INFO:teuthology.orchestra.run.smithi178.stdout:Generating grub configuration file ... 2024-01-11T21:34:16.582 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2024-01-11T21:34:16.618 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2024-01-11T21:34:17.002 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-01-11T21:34:17.014 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-01-11T21:34:19.096 INFO:teuthology.orchestra.run.smithi178.stdout:done 2024-01-11T21:34:19.535 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 20 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta 2024-01-11T21:34:19.575 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.4.0.169.167 2024-01-11T21:34:19.576 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2024-01-11T21:34:19.576 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2024-01-11T21:34:19.576 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2024-01-11T21:34:19.576 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2024-01-11T21:34:19.576 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2024-01-11T21:34:19.576 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2024-01-11T21:34:19.576 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-169-generic 2024-01-11T21:34:19.576 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-11T21:34:19.576 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-11T21:34:19.577 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-11T21:34:19.577 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-01-11T21:34:19.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2024-01-11T21:34:19.664 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-01-11T21:34:19.802 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-01-11T21:34:19.811 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-01-11T21:34:19.831 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-01-11T21:34:19.914 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-01-11T21:34:21.581 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-01-11T21:34:21.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-01-11T21:34:21.670 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-01-11T21:34:21.852 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-01-11T21:34:21.853 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-01-11T21:34:22.070 INFO:teuthology.orchestra.run.smithi178.stdout:linux-image-generic is already the newest version (5.4.0.169.167). 2024-01-11T21:34:22.070 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 0 newly installed, 0 to remove and 282 not upgraded. 2024-01-11T21:34:22.070 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic 2024-01-11T21:34:22.091 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic 2024-01-11T21:34:22.092 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2024-01-11T21:34:22.092 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2024-01-11T21:34:22.092 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2024-01-11T21:34:22.092 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 20 2024-01-11T21:34:22.092 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2024-01-11T21:34:22.092 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2024-01-11T21:34:22.092 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta 2024-01-11T21:34:22.093 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.4.0.169.167 2024-01-11T21:34:22.093 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2024-01-11T21:34:22.093 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2024-01-11T21:34:22.093 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2024-01-11T21:34:22.093 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2024-01-11T21:34:22.094 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2024-01-11T21:34:22.094 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2024-01-11T21:34:22.094 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-01-11T21:34:22.147 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.h256ym8OeZ 2024-01-11T21:34:22.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub/grub.cfg /tmp/tmp.h256ym8OeZ 2024-01-11T21:34:22.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.h256ym8OeZ 2024-01-11T21:34:22.244 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.h256ym8OeZ is 9KB 2024-01-11T21:34:22.466 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.h256ym8OeZ 2024-01-11T21:34:22.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-01-11T21:34:22.530 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:34:22.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-01-11T21:34:22.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-01-11T21:34:22.558 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-169-generic 2024-01-11T21:34:22.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo update-grub 2024-01-11T21:34:23.857 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub' 2024-01-11T21:34:23.865 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-11T21:34:23.885 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-01-11T21:34:24.172 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2024-01-11T21:34:24.183 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2024-01-11T21:34:24.575 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-01-11T21:34:24.587 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-01-11T21:34:25.623 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-01-11T21:34:25.625 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-01-11T21:34:55.656 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-11T21:34:55.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-11T21:34:55.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T21:35:13.984 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-01-11T21:35:22.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-11T21:35:22.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T21:35:26.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-01-11T21:35:38.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-11T21:35:38.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T21:35:41.120 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-01-11T21:35:56.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-11T21:35:56.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T21:36:27.948 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-01-11T21:36:28.548 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-11T21:36:28.548 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-169-generic"... 2024-01-11T21:36:28.548 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-01-11T21:36:28.554 INFO:teuthology.orchestra.run.smithi178.stdout:5.4.0-169-generic 2024-01-11T21:36:28.555 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-169-generic vs 5.4.0-169-generic 2024-01-11T21:36:28.555 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-11T21:36:28.555 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-01-11T21:36:29.556 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-11T21:36:29.556 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-11T21:36:29.661 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-01-11T21:36:29.690 DEBUG:teuthology.parallel:result is None 2024-01-11T21:36:29.690 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-11T21:36:29.697 INFO:teuthology.task.internal:Creating test directory... 2024-01-11T21:36:29.698 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-11T21:36:29.705 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-11T21:36:29.736 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-11T21:36:29.750 INFO:teuthology.task.internal:Creating archive directory... 2024-01-11T21:36:29.750 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-11T21:36:29.772 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-11T21:36:29.781 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-11T21:36:29.781 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-11T21:36:29.828 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T21:36:29.839 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T21:36:29.841 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-11T21:36:29.848 INFO:teuthology.task.internal:Configuring sudo... 2024-01-11T21:36:29.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-11T21:36:29.858 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-11T21:36:29.896 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-11T21:36:29.896 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-11T21:36:29.903 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T21:36:29.950 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T21:36:29.957 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:36:29.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-11T21:36:29.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-01-11T21:36:30.038 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-11T21:36:30.046 INFO:teuthology.task.internal:Starting timer... 2024-01-11T21:36:30.046 INFO:teuthology.run_tasks:Running task pcp... 2024-01-11T21:36:30.057 INFO:teuthology.run_tasks:Running task selinux... 2024-01-11T21:36:30.065 DEBUG:teuthology.task.selinux:Excluding smithi178: OS 'ubuntu' does not support SELinux 2024-01-11T21:36:30.066 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-11T21:36:30.066 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-01-11T21:36:30.066 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-11T21:36:30.066 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-11T21:36:30.077 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-11T21:36:30.094 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-11T21:36:30.096 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-11T21:41:48.162 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-01-11T21:41:48.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-11T21:41:48.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T21:41:48.257 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-01-11T21:41:48.381 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-11T21:41:48.382 INFO:teuthology.run_tasks:Running task clock... 2024-01-11T21:41:48.392 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-11T21:41:48.392 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-11T21:41:48.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T21:41:48.424 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-01-11T21:41:48.424 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Command line: ntpd -gq 2024-01-11T21:41:48.424 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: proto: precision = 0.068 usec (-24) 2024-01-11T21:41:48.425 INFO:teuthology.orchestra.run.smithi178.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-11T21:41:48.425 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-11T21:41:48.425 INFO:teuthology.orchestra.run.smithi178.stderr:restrict ::: KOD does nothing without LIMITED. 2024-01-11T21:41:48.426 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: restrict ::: KOD does nothing without LIMITED. 2024-01-11T21:41:48.426 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Listen and drop on 0 v6wildcard [::]:123 2024-01-11T21:41:48.426 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-01-11T21:41:48.427 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Listen normally on 2 lo 127.0.0.1:123 2024-01-11T21:41:48.427 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Listen normally on 3 enp3s0f1 172.21.15.178:123 2024-01-11T21:41:48.427 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Listen normally on 4 lo [::1]:123 2024-01-11T21:41:48.427 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ef9%5]:123 2024-01-11T21:41:48.427 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:48 ntpd[12732]: Listening on routing socket on fd #22 for interface updates 2024-01-11T21:41:49.424 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:49 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:49.424 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:49 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:50.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:50 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:50.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:50 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:50.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:50 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:50.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:50 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:50.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:50 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:50.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:50 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:51.424 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:51 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:51.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:51 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:51.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:51 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:51.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:51 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:51.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:51 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:51.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:51 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:52.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:52 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:52.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:52 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:52.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:52 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:52.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:52 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:52.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:52 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:52.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:52 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:53.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:53.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:53.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:53 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:54.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:54 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:54.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:54 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:54.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:54 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:54.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:54 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:54.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:54 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:54.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:54 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:55.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:55.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:55.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:55.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:55.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:55.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:55.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:55.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:55.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:55.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:55.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:55.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:55 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: can't open /var/log/ntpstats/rawstats.20240111: Permission denied 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: can't open /var/log/ntpstats/peerstats.20240111: Permission denied 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stdout:11 Jan 21:41:56 ntpd[12732]: ntpd: time slew +0.003542 s 2024-01-11T21:41:56.425 INFO:teuthology.orchestra.run.smithi178.stdout:ntpd: time slew +0.003542s 2024-01-11T21:41:56.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-01-11T21:41:56.426 INFO:teuthology.orchestra.run.smithi178.stderr:11 Jan 21:41:56 ntpd[12732]: can't open /var/log/ntpstats/loopstats.20240111: Permission denied 2024-01-11T21:41:56.485 INFO:teuthology.orchestra.run.smithi178.stdout: remote refid st t when poll reach delay offset jitter 2024-01-11T21:41:56.485 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================== 2024-01-11T21:41:56.485 INFO:teuthology.orchestra.run.smithi178.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-11T21:41:56.485 INFO:teuthology.orchestra.run.smithi178.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-11T21:41:56.485 INFO:teuthology.orchestra.run.smithi178.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-11T21:41:56.486 INFO:teuthology.orchestra.run.smithi178.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-11T21:41:56.486 INFO:teuthology.run_tasks:Running task install... 2024-01-11T21:41:56.494 DEBUG:teuthology.task.install:project ceph 2024-01-11T21:41:56.495 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd900343f97c65fbc993efbb0bd33f08284a8d78b'}} 2024-01-11T21:41:56.495 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd900343f97c65fbc993efbb0bd33f08284a8d78b'} 2024-01-11T21:41:56.495 INFO:teuthology.task.install:Using flavor: default 2024-01-11T21:41:56.500 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-11T21:41:56.500 INFO:teuthology.task.install:extra packages: [] 2024-01-11T21:41:56.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-key list | grep Ceph 2024-01-11T21:41:56.597 INFO:teuthology.orchestra.run.smithi178.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-01-11T21:41:56.658 INFO:teuthology.orchestra.run.smithi178.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-01-11T21:41:56.658 INFO:teuthology.orchestra.run.smithi178.stdout:uid [ unknown] Ceph.com (release key) 2024-01-11T21:41:56.659 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-01-11T21:41:56.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d900343f97c65fbc993efbb0bd33f08284a8d78b 2024-01-11T21:41:56.803 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default/ 2024-01-11T21:41:56.803 INFO:teuthology.task.install.deb:Package version is 17.2.7-234-gd900343f-1focal 2024-01-11T21:41:56.972 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:41:56.972 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-01-11T21:41:56.989 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-01-11T21:41:57.139 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-01-11T21:41:57.156 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-01-11T21:41:57.177 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-01-11T21:41:57.246 INFO:teuthology.orchestra.run.smithi178.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal InRelease 2024-01-11T21:41:57.246 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-01-11T21:41:57.283 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-01-11T21:41:57.320 INFO:teuthology.orchestra.run.smithi178.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal Release.gpg 2024-01-11T21:41:57.357 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-01-11T21:41:57.403 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main i386 Packages [3,364 B] 2024-01-11T21:41:57.791 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 28.4 kB in 1s (44.3 kB/s) 2024-01-11T21:41:58.857 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-01-11T21:41:58.882 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-234-gd900343f-1focal cephadm=17.2.7-234-gd900343f-1focal ceph-mds=17.2.7-234-gd900343f-1focal ceph-mgr=17.2.7-234-gd900343f-1focal ceph-common=17.2.7-234-gd900343f-1focal ceph-fuse=17.2.7-234-gd900343f-1focal ceph-test=17.2.7-234-gd900343f-1focal ceph-volume=17.2.7-234-gd900343f-1focal radosgw=17.2.7-234-gd900343f-1focal python3-rados=17.2.7-234-gd900343f-1focal python3-rgw=17.2.7-234-gd900343f-1focal python3-cephfs=17.2.7-234-gd900343f-1focal python3-rbd=17.2.7-234-gd900343f-1focal libcephfs2=17.2.7-234-gd900343f-1focal libcephfs-dev=17.2.7-234-gd900343f-1focal librados2=17.2.7-234-gd900343f-1focal librbd1=17.2.7-234-gd900343f-1focal rbd-fuse=17.2.7-234-gd900343f-1focal 2024-01-11T21:41:58.951 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-01-11T21:41:59.132 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-01-11T21:41:59.133 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-01-11T21:41:59.310 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages were automatically installed and are no longer required: 2024-01-11T21:41:59.311 INFO:teuthology.orchestra.run.smithi178.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-01-11T21:41:59.311 INFO:teuthology.orchestra.run.smithi178.stdout:Use 'sudo apt autoremove' to remove them. 2024-01-11T21:41:59.311 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2024-01-11T21:41:59.311 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-01-11T21:41:59.312 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-01-11T21:41:59.312 INFO:teuthology.orchestra.run.smithi178.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-01-11T21:41:59.312 INFO:teuthology.orchestra.run.smithi178.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-01-11T21:41:59.312 INFO:teuthology.orchestra.run.smithi178.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-01-11T21:41:59.313 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-01-11T21:41:59.313 INFO:teuthology.orchestra.run.smithi178.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-01-11T21:41:59.314 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: socat xmlstarlet 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout:Recommended packages: 2024-01-11T21:41:59.315 INFO:teuthology.orchestra.run.smithi178.stdout: btrfs-tools 2024-01-11T21:41:59.429 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2024-01-11T21:41:59.430 INFO:teuthology.orchestra.run.smithi178.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-01-11T21:41:59.430 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-01-11T21:41:59.430 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-01-11T21:41:59.430 INFO:teuthology.orchestra.run.smithi178.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-01-11T21:41:59.430 INFO:teuthology.orchestra.run.smithi178.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-01-11T21:41:59.430 INFO:teuthology.orchestra.run.smithi178.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-01-11T21:41:59.430 INFO:teuthology.orchestra.run.smithi178.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-01-11T21:41:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-01-11T21:41:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-01-11T21:41:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-01-11T21:41:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-01-11T21:41:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-01-11T21:41:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-01-11T21:41:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages will be upgraded: 2024-01-11T21:41:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 librbd1 2024-01-11T21:41:59.502 INFO:teuthology.orchestra.run.smithi178.stdout:2 upgraded, 108 newly installed, 0 to remove and 261 not upgraded. 2024-01-11T21:41:59.502 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 210 MB of archives. 2024-01-11T21:41:59.503 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 980 MB of additional disk space will be used. 2024-01-11T21:41:59.503 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-01-11T21:41:59.602 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-01-11T21:41:59.604 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-01-11T21:41:59.615 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-01-11T21:41:59.623 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-01-11T21:41:59.651 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-01-11T21:41:59.652 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-01-11T21:41:59.657 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-234-gd900343f-1focal [3,503 kB] 2024-01-11T21:41:59.660 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-01-11T21:41:59.697 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-01-11T21:41:59.700 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-01-11T21:41:59.706 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-01-11T21:41:59.708 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-01-11T21:41:59.708 INFO:teuthology.orchestra.run.smithi178.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2024-01-11T21:41:59.710 INFO:teuthology.orchestra.run.smithi178.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-01-11T21:41:59.711 INFO:teuthology.orchestra.run.smithi178.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-01-11T21:41:59.711 INFO:teuthology.orchestra.run.smithi178.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-01-11T21:41:59.713 INFO:teuthology.orchestra.run.smithi178.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-01-11T21:41:59.718 INFO:teuthology.orchestra.run.smithi178.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-01-11T21:41:59.734 INFO:teuthology.orchestra.run.smithi178.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-01-11T21:41:59.734 INFO:teuthology.orchestra.run.smithi178.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-01-11T21:41:59.735 INFO:teuthology.orchestra.run.smithi178.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-01-11T21:41:59.736 INFO:teuthology.orchestra.run.smithi178.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-01-11T21:41:59.737 INFO:teuthology.orchestra.run.smithi178.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-01-11T21:41:59.737 INFO:teuthology.orchestra.run.smithi178.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2024-01-11T21:41:59.739 INFO:teuthology.orchestra.run.smithi178.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-01-11T21:41:59.740 INFO:teuthology.orchestra.run.smithi178.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-01-11T21:41:59.741 INFO:teuthology.orchestra.run.smithi178.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-01-11T21:41:59.759 INFO:teuthology.orchestra.run.smithi178.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-01-11T21:41:59.759 INFO:teuthology.orchestra.run.smithi178.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-01-11T21:41:59.765 INFO:teuthology.orchestra.run.smithi178.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-01-11T21:41:59.765 INFO:teuthology.orchestra.run.smithi178.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-01-11T21:41:59.766 INFO:teuthology.orchestra.run.smithi178.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-01-11T21:41:59.779 INFO:teuthology.orchestra.run.smithi178.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-01-11T21:41:59.780 INFO:teuthology.orchestra.run.smithi178.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2024-01-11T21:41:59.783 INFO:teuthology.orchestra.run.smithi178.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2024-01-11T21:41:59.784 INFO:teuthology.orchestra.run.smithi178.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-01-11T21:41:59.785 INFO:teuthology.orchestra.run.smithi178.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-01-11T21:41:59.801 INFO:teuthology.orchestra.run.smithi178.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-01-11T21:41:59.802 INFO:teuthology.orchestra.run.smithi178.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-01-11T21:41:59.803 INFO:teuthology.orchestra.run.smithi178.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-01-11T21:41:59.804 INFO:teuthology.orchestra.run.smithi178.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-01-11T21:41:59.804 INFO:teuthology.orchestra.run.smithi178.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-01-11T21:41:59.909 INFO:teuthology.orchestra.run.smithi178.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-01-11T21:41:59.930 INFO:teuthology.orchestra.run.smithi178.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-01-11T21:41:59.932 INFO:teuthology.orchestra.run.smithi178.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-01-11T21:41:59.949 INFO:teuthology.orchestra.run.smithi178.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-01-11T21:41:59.949 INFO:teuthology.orchestra.run.smithi178.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-01-11T21:41:59.949 INFO:teuthology.orchestra.run.smithi178.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-01-11T21:42:00.004 INFO:teuthology.orchestra.run.smithi178.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-01-11T21:42:00.005 INFO:teuthology.orchestra.run.smithi178.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-01-11T21:42:00.006 INFO:teuthology.orchestra.run.smithi178.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-01-11T21:42:00.007 INFO:teuthology.orchestra.run.smithi178.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-01-11T21:42:00.009 INFO:teuthology.orchestra.run.smithi178.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-01-11T21:42:00.009 INFO:teuthology.orchestra.run.smithi178.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-01-11T21:42:00.010 INFO:teuthology.orchestra.run.smithi178.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-01-11T21:42:00.015 INFO:teuthology.orchestra.run.smithi178.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-01-11T21:42:00.024 INFO:teuthology.orchestra.run.smithi178.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-01-11T21:42:00.038 INFO:teuthology.orchestra.run.smithi178.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-01-11T21:42:00.039 INFO:teuthology.orchestra.run.smithi178.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-01-11T21:42:00.044 INFO:teuthology.orchestra.run.smithi178.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-01-11T21:42:00.048 INFO:teuthology.orchestra.run.smithi178.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-01-11T21:42:00.064 INFO:teuthology.orchestra.run.smithi178.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2024-01-11T21:42:00.067 INFO:teuthology.orchestra.run.smithi178.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-01-11T21:42:00.126 INFO:teuthology.orchestra.run.smithi178.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-01-11T21:42:00.127 INFO:teuthology.orchestra.run.smithi178.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-01-11T21:42:00.127 INFO:teuthology.orchestra.run.smithi178.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-01-11T21:42:00.127 INFO:teuthology.orchestra.run.smithi178.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-01-11T21:42:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-01-11T21:42:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-01-11T21:42:00.142 INFO:teuthology.orchestra.run.smithi178.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-01-11T21:42:00.142 INFO:teuthology.orchestra.run.smithi178.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-01-11T21:42:00.142 INFO:teuthology.orchestra.run.smithi178.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-01-11T21:42:00.147 INFO:teuthology.orchestra.run.smithi178.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-01-11T21:42:00.167 INFO:teuthology.orchestra.run.smithi178.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-01-11T21:42:00.168 INFO:teuthology.orchestra.run.smithi178.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-01-11T21:42:00.169 INFO:teuthology.orchestra.run.smithi178.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-01-11T21:42:00.169 INFO:teuthology.orchestra.run.smithi178.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-01-11T21:42:00.172 INFO:teuthology.orchestra.run.smithi178.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-01-11T21:42:00.173 INFO:teuthology.orchestra.run.smithi178.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-01-11T21:42:00.192 INFO:teuthology.orchestra.run.smithi178.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-234-gd900343f-1focal [3,554 kB] 2024-01-11T21:42:00.446 INFO:teuthology.orchestra.run.smithi178.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-234-gd900343f-1focal [759 kB] 2024-01-11T21:42:00.484 INFO:teuthology.orchestra.run.smithi178.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-234-gd900343f-1focal [413 kB] 2024-01-11T21:42:00.516 INFO:teuthology.orchestra.run.smithi178.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-234-gd900343f-1focal [22.6 kB] 2024-01-11T21:42:00.517 INFO:teuthology.orchestra.run.smithi178.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-234-gd900343f-1focal [200 kB] 2024-01-11T21:42:00.522 INFO:teuthology.orchestra.run.smithi178.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-234-gd900343f-1focal [47.2 kB] 2024-01-11T21:42:00.523 INFO:teuthology.orchestra.run.smithi178.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-234-gd900343f-1focal [385 kB] 2024-01-11T21:42:00.552 INFO:teuthology.orchestra.run.smithi178.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-234-gd900343f-1focal [5,578 kB] 2024-01-11T21:42:00.804 INFO:teuthology.orchestra.run.smithi178.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-234-gd900343f-1focal [116 kB] 2024-01-11T21:42:00.807 INFO:teuthology.orchestra.run.smithi178.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-234-gd900343f-1focal [480 kB] 2024-01-11T21:42:00.815 INFO:teuthology.orchestra.run.smithi178.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-234-gd900343f-1focal [27.8 MB] 2024-01-11T21:42:01.866 INFO:teuthology.orchestra.run.smithi178.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-234-gd900343f-1focal [6,458 kB] 2024-01-11T21:42:01.979 INFO:teuthology.orchestra.run.smithi178.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-234-gd900343f-1focal [221 kB] 2024-01-11T21:42:01.984 INFO:teuthology.orchestra.run.smithi178.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-234-gd900343f-1focal [152 kB] 2024-01-11T21:42:01.990 INFO:teuthology.orchestra.run.smithi178.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-234-gd900343f-1focal [1,589 kB] 2024-01-11T21:42:02.042 INFO:teuthology.orchestra.run.smithi178.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-234-gd900343f-1focal [7,851 kB] 2024-01-11T21:42:02.339 INFO:teuthology.orchestra.run.smithi178.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-234-gd900343f-1focal [28.7 MB] 2024-01-11T21:42:03.406 INFO:teuthology.orchestra.run.smithi178.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-234-gd900343f-1focal [3,940 B] 2024-01-11T21:42:03.406 INFO:teuthology.orchestra.run.smithi178.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-234-gd900343f-1focal [868 kB] 2024-01-11T21:42:03.420 INFO:teuthology.orchestra.run.smithi178.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-234-gd900343f-1focal [2,408 kB] 2024-01-11T21:42:03.437 INFO:teuthology.orchestra.run.smithi178.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-234-gd900343f-1focal [79.2 kB] 2024-01-11T21:42:03.438 INFO:teuthology.orchestra.run.smithi178.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-234-gd900343f-1focal [117 kB] 2024-01-11T21:42:03.439 INFO:teuthology.orchestra.run.smithi178.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-234-gd900343f-1focal [1,760 kB] 2024-01-11T21:42:03.476 INFO:teuthology.orchestra.run.smithi178.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-234-gd900343f-1focal [7,885 kB] 2024-01-11T21:42:03.736 INFO:teuthology.orchestra.run.smithi178.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-234-gd900343f-1focal [4,056 B] 2024-01-11T21:42:03.736 INFO:teuthology.orchestra.run.smithi178.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-234-gd900343f-1focal [56.4 MB] 2024-01-11T21:42:05.896 INFO:teuthology.orchestra.run.smithi178.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-234-gd900343f-1focal [229 kB] 2024-01-11T21:42:05.900 INFO:teuthology.orchestra.run.smithi178.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-234-gd900343f-1focal [21.9 kB] 2024-01-11T21:42:05.901 INFO:teuthology.orchestra.run.smithi178.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-234-gd900343f-1focal [17.1 MB] 2024-01-11T21:42:06.449 INFO:teuthology.orchestra.run.smithi178.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/quincy/d900343f97c65fbc993efbb0bd33f08284a8d78b/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-234-gd900343f-1focal [79.5 kB] 2024-01-11T21:42:07.067 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 210 MB in 7s (29.9 MB/s) 2024-01-11T21:42:07.273 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-01-11T21:42:07.332 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141731 files and directories currently installed.) 2024-01-11T21:42:07.336 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-01-11T21:42:07.421 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-01-11T21:42:07.820 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../001-librbd1_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:07.947 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librbd1 (17.2.7-234-gd900343f-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2024-01-11T21:42:08.680 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../002-librados2_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:08.843 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librados2 (17.2.7-234-gd900343f-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2024-01-11T21:42:09.589 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libcephfs2. 2024-01-11T21:42:09.606 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:09.656 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libcephfs2 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:10.033 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rados. 2024-01-11T21:42:10.051 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../004-python3-rados_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:10.091 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rados (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:10.419 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-01-11T21:42:10.438 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:10.495 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-ceph-argparse (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:10.721 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cephfs. 2024-01-11T21:42:10.739 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:10.780 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cephfs (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:11.159 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-ceph-common. 2024-01-11T21:42:11.179 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:11.224 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-ceph-common (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:11.561 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-prettytable. 2024-01-11T21:42:11.577 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-01-11T21:42:11.619 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-01-11T21:42:11.888 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rbd. 2024-01-11T21:42:11.908 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:11.946 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rbd (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:12.291 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-01-11T21:42:12.309 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-01-11T21:42:12.349 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-01-11T21:42:12.660 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-01-11T21:42:12.679 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-01-11T21:42:12.710 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-01-11T21:42:13.046 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-01-11T21:42:13.064 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-01-11T21:42:13.104 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-01-11T21:42:13.517 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-01-11T21:42:13.534 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-01-11T21:42:13.600 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-01-11T21:42:13.878 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-01-11T21:42:13.894 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-01-11T21:42:13.945 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-01-11T21:42:14.448 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-01-11T21:42:14.467 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-01-11T21:42:14.517 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-01-11T21:42:15.034 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-01-11T21:42:15.052 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-01-11T21:42:15.092 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-01-11T21:42:15.404 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-01-11T21:42:15.419 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-01-11T21:42:15.453 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-01-11T21:42:15.848 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-01-11T21:42:15.868 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-01-11T21:42:15.931 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-01-11T21:42:16.301 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librgw2. 2024-01-11T21:42:16.317 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../019-librgw2_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:16.359 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librgw2 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:17.378 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rgw. 2024-01-11T21:42:17.396 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:17.428 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rgw (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:17.782 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liboath0. 2024-01-11T21:42:17.798 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-01-11T21:42:17.840 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-01-11T21:42:18.176 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libradosstriper1. 2024-01-11T21:42:18.193 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:18.251 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libradosstriper1 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:18.662 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-common. 2024-01-11T21:42:18.680 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../023-ceph-common_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:18.745 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-common (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:21.467 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-base. 2024-01-11T21:42:21.487 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../024-ceph-base_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:21.552 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-base (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:22.647 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cryptography. 2024-01-11T21:42:22.662 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2024-01-11T21:42:22.705 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2024-01-11T21:42:23.117 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-openssl. 2024-01-11T21:42:23.133 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-01-11T21:42:23.200 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-01-11T21:42:23.461 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-01-11T21:42:23.477 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:23.552 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-modules-core (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:23.957 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-bcrypt. 2024-01-11T21:42:23.974 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-01-11T21:42:24.015 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-01-11T21:42:24.335 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cherrypy3. 2024-01-11T21:42:24.350 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2024-01-11T21:42:24.393 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-01-11T21:42:24.780 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-natsort. 2024-01-11T21:42:24.795 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2024-01-11T21:42:24.859 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-01-11T21:42:25.174 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jwt. 2024-01-11T21:42:25.190 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-01-11T21:42:25.249 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-01-11T21:42:25.586 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-logutils. 2024-01-11T21:42:25.602 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2024-01-11T21:42:25.645 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-01-11T21:42:25.972 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-markupsafe. 2024-01-11T21:42:25.987 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-01-11T21:42:26.031 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-01-11T21:42:26.374 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-mako. 2024-01-11T21:42:26.390 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-01-11T21:42:26.433 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-01-11T21:42:26.794 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-simplegeneric. 2024-01-11T21:42:26.810 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2024-01-11T21:42:26.852 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-01-11T21:42:27.180 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-singledispatch. 2024-01-11T21:42:27.197 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-01-11T21:42:27.260 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-01-11T21:42:27.583 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-webob. 2024-01-11T21:42:27.601 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2024-01-11T21:42:27.658 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2024-01-11T21:42:28.011 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-soupsieve. 2024-01-11T21:42:28.029 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-01-11T21:42:28.069 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-01-11T21:42:28.430 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-bs4. 2024-01-11T21:42:28.446 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2024-01-11T21:42:28.488 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-01-11T21:42:28.875 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-waitress. 2024-01-11T21:42:28.893 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-01-11T21:42:28.936 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-01-11T21:42:29.328 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-tempita. 2024-01-11T21:42:29.348 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2024-01-11T21:42:29.386 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-01-11T21:42:29.722 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-paste. 2024-01-11T21:42:29.738 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-01-11T21:42:29.780 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-01-11T21:42:30.125 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-01-11T21:42:30.142 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-01-11T21:42:30.183 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-01-11T21:42:30.511 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pastedeploy. 2024-01-11T21:42:30.527 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2024-01-11T21:42:30.569 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-01-11T21:42:30.880 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-webtest. 2024-01-11T21:42:30.896 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2024-01-11T21:42:30.939 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-01-11T21:42:31.275 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pecan. 2024-01-11T21:42:31.293 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-01-11T21:42:31.333 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-01-11T21:42:31.679 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-werkzeug. 2024-01-11T21:42:31.695 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2024-01-11T21:42:31.771 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-01-11T21:42:32.149 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-01-11T21:42:32.166 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:32.208 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:32.585 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr. 2024-01-11T21:42:32.602 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../049-ceph-mgr_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:32.735 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:33.697 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mon. 2024-01-11T21:42:33.714 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../050-ceph-mon_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:33.746 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mon (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:34.975 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-osd. 2024-01-11T21:42:34.993 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../051-ceph-osd_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:35.033 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-osd (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:37.851 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph. 2024-01-11T21:42:37.872 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../052-ceph_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:37.919 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:38.255 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-fuse. 2024-01-11T21:42:38.273 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../053-ceph-fuse_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:38.321 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-fuse (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:38.825 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mds. 2024-01-11T21:42:38.842 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../054-ceph-mds_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:38.904 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mds (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:39.436 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package cephadm. 2024-01-11T21:42:39.452 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../055-cephadm_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:39.486 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking cephadm (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:39.814 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jinja2. 2024-01-11T21:42:39.830 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2024-01-11T21:42:39.872 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2024-01-11T21:42:40.217 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-01-11T21:42:40.234 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:40.275 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-cephadm (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:40.661 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-repoze.lru. 2024-01-11T21:42:40.678 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2024-01-11T21:42:40.719 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-01-11T21:42:40.997 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libjs-underscore. 2024-01-11T21:42:41.016 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-01-11T21:42:41.057 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-01-11T21:42:41.359 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-01-11T21:42:41.376 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-01-11T21:42:41.418 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-01-11T21:42:41.737 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-routes. 2024-01-11T21:42:41.754 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2024-01-11T21:42:41.795 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-01-11T21:42:42.148 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-01-11T21:42:42.164 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:42.206 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-dashboard (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:42.668 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-decorator. 2024-01-11T21:42:42.688 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-01-11T21:42:42.751 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-01-11T21:42:43.087 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-01-11T21:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-01-11T21:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-01-11T21:42:43.490 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-scipy. 2024-01-11T21:42:43.506 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-01-11T21:42:43.548 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-01-11T21:42:44.890 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-01-11T21:42:44.906 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-01-11T21:42:44.948 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-01-11T21:42:45.502 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-joblib. 2024-01-11T21:42:45.521 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2024-01-11T21:42:45.560 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-01-11T21:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-sklearn. 2024-01-11T21:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-01-11T21:42:46.013 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-01-11T21:42:46.738 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-01-11T21:42:46.754 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:46.795 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:47.982 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cachetools. 2024-01-11T21:42:48.001 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2024-01-11T21:42:48.049 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-01-11T21:42:48.376 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyasn1. 2024-01-11T21:42:48.396 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-01-11T21:42:48.434 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-01-11T21:42:48.929 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-01-11T21:42:48.949 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-01-11T21:42:49.229 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-01-11T21:42:49.557 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rsa. 2024-01-11T21:42:49.577 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2024-01-11T21:42:49.640 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rsa (4.0-3) ... 2024-01-11T21:42:49.927 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-google-auth. 2024-01-11T21:42:49.943 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2024-01-11T21:42:50.011 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-01-11T21:42:50.346 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-blinker. 2024-01-11T21:42:50.364 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-01-11T21:42:50.404 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-01-11T21:42:50.740 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-oauthlib. 2024-01-11T21:42:50.758 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-01-11T21:42:50.798 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-01-11T21:42:51.140 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-01-11T21:42:51.160 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-01-11T21:42:51.198 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-01-11T21:42:51.518 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-websocket. 2024-01-11T21:42:51.534 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-01-11T21:42:51.568 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-01-11T21:42:51.863 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-kubernetes. 2024-01-11T21:42:51.879 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-01-11T21:42:51.921 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-01-11T21:42:52.526 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-01-11T21:42:52.543 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:52.584 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:52.997 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libonig5:amd64. 2024-01-11T21:42:53.016 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2024-01-11T21:42:53.055 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-01-11T21:42:53.476 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libjq1:amd64. 2024-01-11T21:42:53.495 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-01-11T21:42:53.534 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-01-11T21:42:53.812 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package jq. 2024-01-11T21:42:53.832 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-01-11T21:42:53.870 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-01-11T21:42:54.142 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package socat. 2024-01-11T21:42:54.161 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2024-01-11T21:42:54.201 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking socat (1.7.3.3-2) ... 2024-01-11T21:42:54.547 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package xmlstarlet. 2024-01-11T21:42:54.563 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-01-11T21:42:54.604 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-01-11T21:42:54.966 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-test. 2024-01-11T21:42:54.985 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../086-ceph-test_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:42:55.024 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-test (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:42:59.853 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-volume. 2024-01-11T21:42:59.869 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../087-ceph-volume_17.2.7-234-gd900343f-1focal_all.deb ... 2024-01-11T21:42:59.936 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-volume (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:00.284 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package fonts-mathjax. 2024-01-11T21:43:00.300 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-01-11T21:43:00.342 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-01-11T21:43:00.870 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libcephfs-dev. 2024-01-11T21:43:00.887 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:43:00.928 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libcephfs-dev (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:01.348 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package nvme-cli. 2024-01-11T21:43:01.366 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2024-01-11T21:43:01.424 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2024-01-11T21:43:01.786 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libjs-mathjax. 2024-01-11T21:43:01.805 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-01-11T21:43:01.844 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-01-11T21:43:03.489 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-natsort-doc. 2024-01-11T21:43:03.508 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2024-01-11T21:43:03.547 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-01-11T21:43:03.900 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-atomicwrites. 2024-01-11T21:43:03.921 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-01-11T21:43:03.958 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-01-11T21:43:04.286 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-attr. 2024-01-11T21:43:04.303 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2024-01-11T21:43:04.352 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-01-11T21:43:04.698 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-webencodings. 2024-01-11T21:43:04.718 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-01-11T21:43:04.755 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-01-11T21:43:05.092 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-html5lib. 2024-01-11T21:43:05.112 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2024-01-11T21:43:05.150 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-01-11T21:43:05.528 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-01-11T21:43:05.548 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-01-11T21:43:05.586 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-01-11T21:43:06.101 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-packaging. 2024-01-11T21:43:06.121 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2024-01-11T21:43:06.159 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-packaging (20.3-1) ... 2024-01-11T21:43:06.495 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pastescript. 2024-01-11T21:43:06.514 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-01-11T21:43:06.553 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-01-11T21:43:06.873 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pluggy. 2024-01-11T21:43:06.893 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-01-11T21:43:06.931 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-01-11T21:43:07.268 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-psutil. 2024-01-11T21:43:07.285 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-01-11T21:43:07.325 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-01-11T21:43:07.673 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-py. 2024-01-11T21:43:07.693 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-01-11T21:43:07.730 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-01-11T21:43:08.076 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyinotify. 2024-01-11T21:43:08.094 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-01-11T21:43:08.158 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-01-11T21:43:08.479 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-wcwidth. 2024-01-11T21:43:08.496 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-01-11T21:43:08.554 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-01-11T21:43:08.878 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pytest. 2024-01-11T21:43:08.896 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-01-11T21:43:08.936 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-01-11T21:43:09.308 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-simplejson. 2024-01-11T21:43:09.325 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-01-11T21:43:09.366 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-01-11T21:43:09.704 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-01-11T21:43:09.723 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-01-11T21:43:09.762 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-01-11T21:43:10.584 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package radosgw. 2024-01-11T21:43:10.601 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../108-radosgw_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:43:10.642 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking radosgw (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:12.363 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package rbd-fuse. 2024-01-11T21:43:12.380 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-234-gd900343f-1focal_amd64.deb ... 2024-01-11T21:43:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking rbd-fuse (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:12.751 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-attr (19.3.0-2) ... 2024-01-11T21:43:13.043 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-01-11T21:43:13.168 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-01-11T21:43:13.328 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2024-01-11T21:43:13.530 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up cephadm (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:14.125 INFO:teuthology.orchestra.run.smithi178.stdout:Adding system user cephadm....done 2024-01-11T21:43:14.286 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-01-11T21:43:14.437 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-01-11T21:43:14.798 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-01-11T21:43:15.091 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-01-11T21:43:15.216 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-01-11T21:43:15.368 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-01-11T21:43:15.720 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-01-11T21:43:16.133 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-01-11T21:43:16.435 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-01-11T21:43:16.753 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-01-11T21:43:17.054 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-01-11T21:43:17.373 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-ceph-argparse (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:17.656 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-01-11T21:43:17.807 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-01-11T21:43:18.120 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-01-11T21:43:18.429 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-01-11T21:43:18.556 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-01-11T21:43:18.924 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-01-11T21:43:19.216 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-01-11T21:43:19.501 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-01-11T21:43:20.344 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-01-11T21:43:20.637 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2024-01-11T21:43:21.040 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-01-11T21:43:21.333 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-packaging (20.3-1) ... 2024-01-11T21:43:21.634 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-01-11T21:43:21.761 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-01-11T21:43:21.887 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-01-11T21:43:22.180 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-01-11T21:43:22.582 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-01-11T21:43:22.958 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2024-01-11T21:43:23.302 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up socat (1.7.3.3-2) ... 2024-01-11T21:43:23.436 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-ceph-common (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:23.755 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-01-11T21:43:23.897 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-01-11T21:43:24.249 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-01-11T21:43:24.408 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-01-11T21:43:24.534 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-01-11T21:43:24.827 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-01-11T21:43:25.145 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-01-11T21:43:25.280 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-01-11T21:43:25.431 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-01-11T21:43:25.766 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-01-11T21:43:26.067 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-01-11T21:43:26.376 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-01-11T21:43:26.502 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-01-11T21:43:26.778 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-01-11T21:43:27.088 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-01-11T21:43:27.398 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-01-11T21:43:27.788 INFO:teuthology.orchestra.run.smithi178.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-01-11T21:43:28.059 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-01-11T21:43:28.185 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-01-11T21:43:28.519 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-01-11T21:43:28.896 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-01-11T21:43:29.022 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-01-11T21:43:29.165 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-webob (1:1.8.5-2) ... 2024-01-11T21:43:29.491 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-01-11T21:43:29.792 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librados2 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:29.918 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-01-11T21:43:31.408 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libsqlite3-mod-ceph (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:31.534 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-01-11T21:43:31.660 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-01-11T21:43:32.111 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-01-11T21:43:32.471 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-01-11T21:43:32.764 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-01-11T21:43:32.907 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-01-11T21:43:33.250 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libcephfs2 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:33.376 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-01-11T21:43:33.744 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libradosstriper1 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:33.896 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-01-11T21:43:34.230 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-01-11T21:43:35.277 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librbd1 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:35.412 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-01-11T21:43:35.571 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-01-11T21:43:35.697 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-01-11T21:43:35.823 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rsa (4.0-3) ... 2024-01-11T21:43:36.150 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-01-11T21:43:36.301 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-01-11T21:43:36.619 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-modules-core (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:36.740 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-fuse (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:36.968 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-01-11T21:43:36.968 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-01-11T21:43:37.416 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libcephfs-dev (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:37.669 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-01-11T21:43:37.971 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-01-11T21:43:38.289 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-01-11T21:43:38.650 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rados (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:38.776 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librgw2 (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:38.928 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rbd (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:39.062 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up rbd-fuse (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:39.214 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-routes (2.4.1-2) ... 2024-01-11T21:43:39.534 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-01-11T21:43:39.835 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rgw (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:39.986 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cephfs (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:40.113 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-01-11T21:43:41.050 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-common (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:41.646 INFO:teuthology.orchestra.run.smithi178.stdout:Adding group ceph....done 2024-01-11T21:43:42.006 INFO:teuthology.orchestra.run.smithi178.stdout:Adding system user ceph....done 2024-01-11T21:43:42.201 INFO:teuthology.orchestra.run.smithi178.stdout:Setting system user ceph properties....done 2024-01-11T21:43:42.226 INFO:teuthology.orchestra.run.smithi178.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-01-11T21:43:42.374 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-01-11T21:43:42.687 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-01-11T21:43:43.145 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-test (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:43.296 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up radosgw (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:43.536 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-01-11T21:43:43.536 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-01-11T21:43:44.199 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-base (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:44.570 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-01-11T21:43:45.017 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mds (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:45.255 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-01-11T21:43:45.256 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-01-11T21:43:45.736 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:45.969 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-01-11T21:43:45.970 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-01-11T21:43:46.421 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-osd (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:46.938 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-01-11T21:43:46.938 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-01-11T21:43:47.381 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-k8sevents (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:47.532 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:47.675 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mon (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:47.904 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-01-11T21:43:47.904 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-01-11T21:43:48.343 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-cephadm (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:48.510 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:48.636 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-dashboard (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:48.813 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-volume (17.2.7-234-gd900343f-1focal) ... 2024-01-11T21:43:48.940 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-01-11T21:43:49.091 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-01-11T21:43:49.393 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-01-11T21:43:52.230 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-01-11T21:43:52.818 INFO:teuthology.orchestra.run.smithi178.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-01-11T21:43:52.823 DEBUG:teuthology.parallel:result is None 2024-01-11T21:43:52.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d900343f97c65fbc993efbb0bd33f08284a8d78b 2024-01-11T21:43:52.964 DEBUG:teuthology.orchestra.run.smithi178:> dpkg-query -W -f '${Version}' ceph 2024-01-11T21:43:52.988 INFO:teuthology.orchestra.run.smithi178.stdout:17.2.7-234-gd900343f-1focal 2024-01-11T21:43:52.988 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-234-gd900343f-1focal 2024-01-11T21:43:52.988 INFO:teuthology.task.install:The correct ceph version 17.2.7-234-gd900343f-1focal is installed. 2024-01-11T21:43:52.990 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-11T21:43:52.990 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:52.990 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-11T21:43:53.045 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-11T21:43:53.045 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-01-11T21:43:53.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-11T21:43:53.071 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-11T21:43:53.072 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-11T21:43:53.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-11T21:43:53.142 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-11T21:43:53.142 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-01-11T21:43:53.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-11T21:43:53.163 INFO:teuthology.run_tasks:Running task ceph... 2024-01-11T21:43:53.229 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-11T21:43:53.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /var/log/ceph 2024-01-11T21:43:53.245 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-11T21:43:53.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-11T21:43:53.301 INFO:tasks.ceph:Creating extra log directories... 2024-01-11T21:43:53.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-11T21:43:53.321 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-11T21:43:53.321 INFO:tasks.ceph:config {'conf': {'client': {'rbd default format': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-11T21:43:53.321 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820', 'branch': 'quincy', 'description': 'rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}', 'email': None, 'first_in_suite': False, 'job_id': '7513820', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'quincy'}, 'ceph': {'conf': {'client': {'rbd default format': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'sha1': 'd900343f97c65fbc993efbb0bd33f08284a8d78b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd900343f97c65fbc993efbb0bd33f08284a8d78b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-63937-quincy', 'sha1': '49908e15c7044e78475683aa6ccd77bd187be2f4'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 7078, 'sha1': 'd900343f97c65fbc993efbb0bd33f08284a8d78b', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rbd', 'suite_branch': 'wip-63937-quincy', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': '49908e15c7044e78475683aa6ccd77bd187be2f4', 'targets': {'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create cache 4', 'sudo ceph osd tier add rbd cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay rbd cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'workunit': {'clients': {'client.0': ['rbd/cli_generic.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-11_21:24:20', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-11T21:43:53.322 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-11T21:43:53.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2024-01-11T21:43:53.344 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.344 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-01-11T21:43:53.351 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-11T21:43:53.351 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-01-11T21:43:53.401 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-11T21:43:53.402 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T21:43:53.402 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-01-11T21:43:53.402 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T21:43:53.402 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-01-11 21:40:21.597232560 +0000 2024-01-11T21:43:53.402 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-01-11 21:40:21.593232653 +0000 2024-01-11T21:43:53.402 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-01-11 21:40:21.593232653 +0000 2024-01-11T21:43:53.402 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-01-11T21:43:53.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-11T21:43:53.416 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-01-11T21:43:53.416 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-01-11T21:43:53.416 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000312496 s, 1.6 MB/s 2024-01-11T21:43:53.418 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-11T21:43:53.425 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 368 Links: 1 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-01-11 21:40:22.229217982 +0000 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-01-11 21:40:22.009223056 +0000 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-01-11 21:40:22.009223056 +0000 2024-01-11T21:43:53.474 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-01-11T21:43:53.474 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-11T21:43:53.489 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-01-11T21:43:53.489 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-01-11T21:43:53.489 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000291131 s, 1.8 MB/s 2024-01-11T21:43:53.490 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-11T21:43:53.499 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-01-11 21:40:22.645208387 +0000 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-01-11 21:40:22.409213831 +0000 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-01-11 21:40:22.409213831 +0000 2024-01-11T21:43:53.550 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-01-11T21:43:53.550 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-11T21:43:53.565 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-01-11T21:43:53.566 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-01-11T21:43:53.566 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000291691 s, 1.8 MB/s 2024-01-11T21:43:53.567 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-11T21:43:53.574 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-01-11T21:43:53.625 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-11T21:43:53.626 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T21:43:53.626 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-01-11T21:43:53.626 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T21:43:53.626 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-01-11 21:40:23.061198790 +0000 2024-01-11T21:43:53.626 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-01-11 21:40:22.869203219 +0000 2024-01-11T21:43:53.626 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-01-11 21:40:22.869203219 +0000 2024-01-11T21:43:53.626 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-01-11T21:43:53.626 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-11T21:43:53.637 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-01-11T21:43:53.637 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-01-11T21:43:53.637 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000290758 s, 1.8 MB/s 2024-01-11T21:43:53.638 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-11T21:43:53.646 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-11T21:43:53.646 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-11T21:43:53.646 INFO:tasks.ceph:Generating config... 2024-01-11T21:43:53.647 INFO:tasks.ceph:[client] rbd default format = 1 2024-01-11T21:43:53.647 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-01-11T21:43:53.647 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-01-11T21:43:53.648 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-01-11T21:43:53.648 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-11T21:43:53.648 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-11T21:43:53.648 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-11T21:43:53.648 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-11T21:43:53.648 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-11T21:43:53.648 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-01-11T21:43:53.648 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-01-11T21:43:53.648 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-11T21:43:53.648 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-11T21:43:53.648 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-11T21:43:53.648 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-11T21:43:53.649 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-11T21:43:53.649 INFO:tasks.ceph:Setting up mon.a... 2024-01-11T21:43:53.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-11T21:43:53.718 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.keyring 2024-01-11T21:43:53.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-11T21:43:53.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T21:43:53.762 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.178')] 2024-01-11T21:43:53.763 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.178', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default format': 1}, 'mon.a': {}} 2024-01-11T21:43:53.763 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.764 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-11T21:43:53.769 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.178 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-11T21:43:53.842 INFO:teuthology.orchestra.run.smithi178.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-11T21:43:53.842 INFO:teuthology.orchestra.run.smithi178.stdout:monmaptool: generated fsid f4a6b9d5-8d9a-4520-875a-7dbacea84608 2024-01-11T21:43:53.842 INFO:teuthology.orchestra.run.smithi178.stdout:setting min_mon_release = octopus 2024-01-11T21:43:53.842 INFO:teuthology.orchestra.run.smithi178.stdout:epoch 0 2024-01-11T21:43:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:fsid f4a6b9d5-8d9a-4520-875a-7dbacea84608 2024-01-11T21:43:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:last_changed 2024-01-11T21:43:53.843275+0000 2024-01-11T21:43:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:created 2024-01-11T21:43:53.843275+0000 2024-01-11T21:43:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:min_mon_release 15 (octopus) 2024-01-11T21:43:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:election_strategy: 1 2024-01-11T21:43:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:0: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.a 2024-01-11T21:43:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-11T21:43:53.844 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-11T21:43:53.850 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f4a6b9d5-8d9a-4520-875a-7dbacea84608... 2024-01-11T21:43:53.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-11T21:43:53.883 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-01-11T21:43:53.883 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-01-11T21:43:53.883 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-01-11T21:43:53.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-01-11T21:43:53.885 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = 172.21.15.178 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: mon client directed command retry = 5 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on pool no app = False 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: ms inject socket failures = 5000 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = f4a6b9d5-8d9a-4520-875a-7dbacea84608 2024-01-11T21:43:53.886 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-01-11T21:43:53.887 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: bluefs allocator = hybrid 2024-01-11T21:43:53.888 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = hybrid 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: osd shutdown pgref assert = True 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-01-11T21:43:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-01-11T21:43:53.890 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.891 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout: rbd default format = 1 2024-01-11T21:43:53.892 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-01-11T21:43:53.893 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-11T21:43:53.893 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-11T21:43:53.982 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-11T21:43:53.982 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.982 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-11T21:43:53.987 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.987 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-11T21:43:53.993 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2024-01-11T21:43:53.994 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:53.994 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-11T21:43:53.994 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T21:43:54.060 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:43:54.060 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-11T21:43:54.068 INFO:tasks.ceph:Setting up mon nodes... 2024-01-11T21:43:54.068 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-11T21:43:54.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-11T21:43:54.111 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-11T21:43:54.113 INFO:tasks.ceph:Setting up mds nodes... 2024-01-11T21:43:54.113 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-11T21:43:54.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-11T21:43:54.191 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-11T21:43:54.204 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-11T21:43:54.204 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-11T21:43:54.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-11T21:43:54.215 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-11T21:43:54.215 INFO:tasks.ceph:role: osd.0 2024-01-11T21:43:54.215 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com 2024-01-11T21:43:54.216 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-11T21:43:54.255 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T21:43:54.256 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T21:43:54.258 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-01-11T21:43:54.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-11T21:43:54.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-11T21:43:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T21:43:54.289 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: /sbin/restorecon: command not found 2024-01-11T21:43:54.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-11T21:43:54.344 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-11T21:43:54.345 INFO:tasks.ceph:role: osd.1 2024-01-11T21:43:54.345 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com 2024-01-11T21:43:54.345 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-11T21:43:54.383 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T21:43:54.384 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T21:43:54.385 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-01-11T21:43:54.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-11T21:43:54.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-11T21:43:54.413 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: /sbin/restorecon: command not found 2024-01-11T21:43:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T21:43:54.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-11T21:43:54.471 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-11T21:43:54.471 INFO:tasks.ceph:role: osd.2 2024-01-11T21:43:54.471 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com 2024-01-11T21:43:54.472 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-11T21:43:54.515 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T21:43:54.515 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T21:43:54.516 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T21:43:54.517 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-01-11T21:43:54.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-11T21:43:54.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-11T21:43:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T21:43:54.543 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: /sbin/restorecon: command not found 2024-01-11T21:43:54.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T21:43:54.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:54.663+0000 7fa8172cc3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-11T21:43:54.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:54.663+0000 7fa8172cc3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-11T21:43:54.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:54.663+0000 7fa8172cc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-11T21:43:54.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:54.663+0000 7fa8172cc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-11T21:43:54.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:54.663+0000 7fa8172cc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-11T21:43:54.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:54.663+0000 7fa8172cc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-11T21:43:55.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:55.227+0000 7fa8172cc3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:43:56.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:55.999+0000 7fa8172cc3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:43:56.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:56.811+0000 7fa8172cc3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:43:57.680 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-11T21:43:57.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T21:43:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:57.731+0000 7f49504943c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-11T21:43:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:57.731+0000 7f49504943c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-11T21:43:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:57.731+0000 7f49504943c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-11T21:43:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:57.731+0000 7f49504943c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-11T21:43:57.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:57.731+0000 7f49504943c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-11T21:43:57.735 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:57.731+0000 7f49504943c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-11T21:43:58.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:58.291+0000 7f49504943c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:43:59.079 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:59.079+0000 7f49504943c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:43:59.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:43:59.855+0000 7f49504943c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:44:00.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-11T21:44:00.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T21:44:00.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:00.803+0000 7f775b9763c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-11T21:44:00.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:00.803+0000 7f775b9763c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-11T21:44:00.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:00.803+0000 7f775b9763c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-11T21:44:00.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:00.803+0000 7f775b9763c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-11T21:44:00.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:00.803+0000 7f775b9763c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-11T21:44:00.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:00.807+0000 7f775b9763c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-11T21:44:01.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:01.367+0000 7f775b9763c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:44:02.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:02.163+0000 7f775b9763c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:44:02.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-11T21:44:02.951+0000 7f775b9763c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T21:44:03.823 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-11T21:44:03.883 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-11T21:44:03.884 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:03.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-11T21:44:03.898 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:03.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-11T21:44:03.910 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:03.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-11T21:44:03.965 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:03.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-11T21:44:03.977 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:03.977 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-11T21:44:03.982 INFO:tasks.ceph:Adding keys to all mons... 2024-01-11T21:44:03.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-11T21:44:03.992 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.x] 2024-01-11T21:44:03.992 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAaYaBlWE2uBhAA94ZbOsaCYVUANsNlri+QFA== 2024-01-11T21:44:03.992 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-01-11T21:44:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAaYaBl6q2gJxAArP6yKniNFZ8BcFkJ1xp54A== 2024-01-11T21:44:03.993 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-01-11T21:44:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAdYaBlIgXPKxAAZx93CdqZWC8ve04NfTbKLQ== 2024-01-11T21:44:03.993 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-01-11T21:44:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAgYaBlwl4jMBAAwUSJ6y60Y3QRhXRinvUlew== 2024-01-11T21:44:03.993 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-01-11T21:44:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAaYaBlDDZ0CxAA+2ZPQmq7jHxjS3plgdQbpw== 2024-01-11T21:44:03.994 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-11T21:44:04.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T21:44:04.121 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T21:44:04.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T21:44:04.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T21:44:04.245 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-11T21:44:04.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-11T21:44:04.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-11T21:44:04.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-11T21:44:04.369 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-11T21:44:04.374 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-11T21:44:04.374 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-11T21:44:04.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-11T21:44:04.376 INFO:tasks.ceph.mon.a:Started 2024-01-11T21:44:04.376 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-11T21:44:04.377 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-11T21:44:04.377 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-11T21:44:04.379 INFO:tasks.ceph.mgr.x:Started 2024-01-11T21:44:04.379 INFO:tasks.ceph:Setting crush tunables to default 2024-01-11T21:44:04.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd crush tunables default 2024-01-11T21:44:04.592 INFO:teuthology.orchestra.run.smithi178.stderr:adjusted tunables profile to default 2024-01-11T21:44:04.602 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-11T21:44:04.602 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:04.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-11T21:44:04.617 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:04.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-11T21:44:04.627 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:44:04.627 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-11T21:44:04.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8 0 2024-01-11T21:44:04.707 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:04.707+0000 7f005bf2f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T21:44:04.832 INFO:teuthology.orchestra.run.smithi178.stdout:0 2024-01-11T21:44:04.844 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new ceb020ed-023b-4e95-8ca5-6b2accf2a58f 1 2024-01-11T21:44:05.027 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-01-11T21:44:05.040 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new dc40179d-412f-444c-b8ca-08f7873edc3d 2 2024-01-11T21:44:05.234 INFO:teuthology.orchestra.run.smithi178.stdout:2 2024-01-11T21:44:05.247 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-11T21:44:05.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-11T21:44:05.250 INFO:tasks.ceph.osd.0:Started 2024-01-11T21:44:05.250 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-11T21:44:05.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-11T21:44:05.254 INFO:tasks.ceph.osd.1:Started 2024-01-11T21:44:05.254 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-11T21:44:05.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-11T21:44:05.256 INFO:tasks.ceph.osd.2:Started 2024-01-11T21:44:05.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T21:44:05.360 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:05.359+0000 7f005bf2f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T21:44:05.416 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:05.415+0000 7f005bf2f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T21:44:05.428 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:44:05.429 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":5,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T21:44:05.231824+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T21:44:05.440 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-11T21:44:05.440 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-11T21:44:05.531 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:05.531+0000 7f005bf2f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T21:44:05.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:44:05.627+0000 7fdf934203c0 -1 Falling back to public interface 2024-01-11T21:44:05.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:44:05.627+0000 7f6c411c73c0 -1 Falling back to public interface 2024-01-11T21:44:05.652 INFO:tasks.ceph.osd.2.smithi178.stderr:2024-01-11T21:44:05.651+0000 7ff92b57d3c0 -1 Falling back to public interface 2024-01-11T21:44:05.743 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T21:44:05.755 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:05.751+0000 7f005bf2f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T21:44:05.804 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:05.803+0000 7f005bf2f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T21:44:05.910 INFO:teuthology.misc.health.smithi178.stdout: 2024-01-11T21:44:05.910 INFO:teuthology.misc.health.smithi178.stdout:{"epoch":5,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T21:44:05.231824+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T21:44:05.921 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-11T21:44:05.980 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:05.979+0000 7f005bf2f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T21:44:06.040 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:06.039+0000 7f005bf2f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T21:44:06.101 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:06.099+0000 7f005bf2f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T21:44:06.498 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:06.495+0000 7f005bf2f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T21:44:06.610 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:06.607+0000 7f005bf2f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T21:44:06.901 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:06.899+0000 7f005bf2f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T21:44:07.065 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.063+0000 7f005bf2f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T21:44:07.130 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.127+0000 7f005bf2f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T21:44:07.182 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.179+0000 7f005bf2f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T21:44:07.238 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.235+0000 7f005bf2f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T21:44:07.524 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.523+0000 7f005bf2f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T21:44:07.580 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.579+0000 7f005bf2f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T21:44:07.645 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.643+0000 7f005bf2f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T21:44:07.805 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.803+0000 7f005bf2f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T21:44:07.870 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:07.867+0000 7f005bf2f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T21:44:08.583 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:08.583+0000 7f005bf2f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T21:44:09.498 INFO:tasks.ceph.osd.2.smithi178.stderr:2024-01-11T21:44:09.495+0000 7ff92b57d3c0 -1 osd.2 0 log_to_monitors true 2024-01-11T21:44:09.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:44:09.511+0000 7f6c411c73c0 -1 osd.0 0 log_to_monitors true 2024-01-11T21:44:09.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:44:09.515+0000 7fdf934203c0 -1 osd.1 0 log_to_monitors true 2024-01-11T21:44:09.822 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-01-11T21:44:09.819+0000 7f0045027700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-11T21:44:12.224 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T21:44:12.670 INFO:teuthology.misc.health.smithi178.stdout: 2024-01-11T21:44:12.670 INFO:teuthology.misc.health.smithi178.stdout:{"epoch":7,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T21:44:11.465574+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T21:44:12.688 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-11T21:44:12.946 INFO:tasks.ceph.osd.2.smithi178.stderr:2024-01-11T21:44:12.943+0000 7ff926d1d700 -1 osd.2 0 waiting for initial osdmap 2024-01-11T21:44:12.954 INFO:tasks.ceph.osd.2.smithi178.stderr:2024-01-11T21:44:12.951+0000 7ff91fac1700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T21:44:12.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:44:12.959+0000 7fdf903e5700 -1 osd.1 0 waiting for initial osdmap 2024-01-11T21:44:12.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:44:12.967+0000 7fdf87964700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T21:44:13.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:44:13.011+0000 7f6c3c967700 -1 osd.0 0 waiting for initial osdmap 2024-01-11T21:44:13.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:44:13.019+0000 7f6c3570b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T21:44:18.990 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T21:44:19.264 INFO:teuthology.misc.health.smithi178.stdout: 2024-01-11T21:44:19.264 INFO:teuthology.misc.health.smithi178.stdout:{"epoch":11,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T21:44:17.468722+0000","last_up_change":"2024-01-11T21:44:13.948237+0000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T21:44:14.674056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20057},{"type":"v1","addr":"172.21.15.178:6806","nonce":20057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20057},{"type":"v1","addr":"172.21.15.178:6810","nonce":20057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20057},{"type":"v1","addr":"172.21.15.178:6815","nonce":20057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20057},{"type":"v1","addr":"172.21.15.178:6813","nonce":20057}]},"public_addr":"172.21.15.178:6806/20057","cluster_addr":"172.21.15.178:6810/20057","heartbeat_back_addr":"172.21.15.178:6815/20057","heartbeat_front_addr":"172.21.15.178:6813/20057","state":["exists","up"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20056},{"type":"v1","addr":"172.21.15.178:6801","nonce":20056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20056},{"type":"v1","addr":"172.21.15.178:6803","nonce":20056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":20056},{"type":"v1","addr":"172.21.15.178:6811","nonce":20056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6805","nonce":20056},{"type":"v1","addr":"172.21.15.178:6807","nonce":20056}]},"public_addr":"172.21.15.178:6801/20056","cluster_addr":"172.21.15.178:6803/20056","heartbeat_back_addr":"172.21.15.178:6811/20056","heartbeat_front_addr":"172.21.15.178:6807/20056","state":["exists","up"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20058},{"type":"v1","addr":"172.21.15.178:6817","nonce":20058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20058},{"type":"v1","addr":"172.21.15.178:6819","nonce":20058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20058},{"type":"v1","addr":"172.21.15.178:6823","nonce":20058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20058},{"type":"v1","addr":"172.21.15.178:6821","nonce":20058}]},"public_addr":"172.21.15.178:6817/20058","cluster_addr":"172.21.15.178:6819/20058","heartbeat_back_addr":"172.21.15.178:6823/20058","heartbeat_front_addr":"172.21.15.178:6821/20058","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.562572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.495598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.468222+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T21:44:19.277 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-11T21:44:19.277 INFO:tasks.ceph:Creating RBD pool 2024-01-11T21:44:19.278 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-11T21:44:20.484 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'rbd' created 2024-01-11T21:44:20.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-11T21:44:21.492 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-11T21:44:21.504 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-11T21:44:21.504 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-11T21:44:21.504 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-11T21:44:21.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-11T21:44:21.505 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-11T21:44:21.834 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:44:21.848 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":19874},{"type":"v1","addr":"172.21.15.178:6825","nonce":19874}]},"active_addr":"172.21.15.178:6825/19874","active_change":"2024-01-11T21:44:08.606584+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1499195295}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":2428781577}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1458107585}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1678011529}]}]} 2024-01-11T21:44:21.850 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-11T21:44:21.850 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-11T21:44:21.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T21:44:22.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:44:22.138 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":13,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T21:44:21.469423+0000","last_up_change":"2024-01-11T21:44:13.948237+0000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T21:44:14.674056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-11T21:44:19.605189+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20057},{"type":"v1","addr":"172.21.15.178:6806","nonce":20057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20057},{"type":"v1","addr":"172.21.15.178:6810","nonce":20057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20057},{"type":"v1","addr":"172.21.15.178:6815","nonce":20057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20057},{"type":"v1","addr":"172.21.15.178:6813","nonce":20057}]},"public_addr":"172.21.15.178:6806/20057","cluster_addr":"172.21.15.178:6810/20057","heartbeat_back_addr":"172.21.15.178:6815/20057","heartbeat_front_addr":"172.21.15.178:6813/20057","state":["exists","up"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20056},{"type":"v1","addr":"172.21.15.178:6801","nonce":20056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20056},{"type":"v1","addr":"172.21.15.178:6803","nonce":20056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":20056},{"type":"v1","addr":"172.21.15.178:6811","nonce":20056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6805","nonce":20056},{"type":"v1","addr":"172.21.15.178:6807","nonce":20056}]},"public_addr":"172.21.15.178:6801/20056","cluster_addr":"172.21.15.178:6803/20056","heartbeat_back_addr":"172.21.15.178:6811/20056","heartbeat_front_addr":"172.21.15.178:6807/20056","state":["exists","up"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20058},{"type":"v1","addr":"172.21.15.178:6817","nonce":20058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20058},{"type":"v1","addr":"172.21.15.178:6819","nonce":20058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20058},{"type":"v1","addr":"172.21.15.178:6823","nonce":20058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20058},{"type":"v1","addr":"172.21.15.178:6821","nonce":20058}]},"public_addr":"172.21.15.178:6817/20058","cluster_addr":"172.21.15.178:6819/20058","heartbeat_back_addr":"172.21.15.178:6823/20058","heartbeat_front_addr":"172.21.15.178:6821/20058","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.562572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.495598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.468222+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T21:44:22.151 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-11T21:44:22.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T21:44:22.437 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:44:22.437 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":13,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T21:44:21.469423+0000","last_up_change":"2024-01-11T21:44:13.948237+0000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T21:44:14.674056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-11T21:44:19.605189+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20057},{"type":"v1","addr":"172.21.15.178:6806","nonce":20057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20057},{"type":"v1","addr":"172.21.15.178:6810","nonce":20057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20057},{"type":"v1","addr":"172.21.15.178:6815","nonce":20057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20057},{"type":"v1","addr":"172.21.15.178:6813","nonce":20057}]},"public_addr":"172.21.15.178:6806/20057","cluster_addr":"172.21.15.178:6810/20057","heartbeat_back_addr":"172.21.15.178:6815/20057","heartbeat_front_addr":"172.21.15.178:6813/20057","state":["exists","up"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20056},{"type":"v1","addr":"172.21.15.178:6801","nonce":20056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20056},{"type":"v1","addr":"172.21.15.178:6803","nonce":20056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":20056},{"type":"v1","addr":"172.21.15.178:6811","nonce":20056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6805","nonce":20056},{"type":"v1","addr":"172.21.15.178:6807","nonce":20056}]},"public_addr":"172.21.15.178:6801/20056","cluster_addr":"172.21.15.178:6803/20056","heartbeat_back_addr":"172.21.15.178:6811/20056","heartbeat_front_addr":"172.21.15.178:6807/20056","state":["exists","up"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20058},{"type":"v1","addr":"172.21.15.178:6817","nonce":20058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20058},{"type":"v1","addr":"172.21.15.178:6819","nonce":20058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20058},{"type":"v1","addr":"172.21.15.178:6823","nonce":20058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20058},{"type":"v1","addr":"172.21.15.178:6821","nonce":20058}]},"public_addr":"172.21.15.178:6817/20058","cluster_addr":"172.21.15.178:6819/20058","heartbeat_back_addr":"172.21.15.178:6823/20058","heartbeat_front_addr":"172.21.15.178:6821/20058","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.562572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.495598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.468222+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T21:44:22.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-11T21:44:22.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-11T21:44:22.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-11T21:44:22.584 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-01-11T21:44:22.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T21:44:22.594 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-01-11T21:44:22.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T21:44:22.598 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-01-11T21:44:22.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T21:44:22.863 INFO:teuthology.orchestra.run.smithi178.stdout:34359738371 2024-01-11T21:44:22.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2024-01-11T21:44:22.882 INFO:teuthology.orchestra.run.smithi178.stdout:34359738371 2024-01-11T21:44:22.894 INFO:teuthology.orchestra.run.smithi178.stdout:34359738371 2024-01-11T21:44:22.895 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2024-01-11T21:44:22.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2024-01-11T21:44:23.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T21:44:23.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T21:44:23.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T21:44:24.183 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-01-11T21:44:24.196 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2024-01-11T21:44:24.196 DEBUG:teuthology.parallel:result is None 2024-01-11T21:44:24.221 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-01-11T21:44:24.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2024-01-11T21:44:24.234 DEBUG:teuthology.parallel:result is None 2024-01-11T21:44:24.277 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-01-11T21:44:24.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2024-01-11T21:44:24.290 DEBUG:teuthology.parallel:result is None 2024-01-11T21:44:24.291 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-11T21:44:24.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T21:44:24.567 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:44:24.570 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-11T21:44:24.581 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-01-11T21:44:22.617956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":905652,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":904128,"kb_avail":282209868,"statfs":{"total":289910292480,"available":288982904832,"internally_reserved":0,"allocated":1536000,"data_stored":1276918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":925827072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.148286"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491077+0000","last_change":"2024-01-11T21:44:21.487885+0000","last_active":"2024-01-11T21:44:21.491077+0000","last_peered":"2024-01-11T21:44:21.491077+0000","last_clean":"2024-01-11T21:44:21.491077+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T21:44:21.491077+0000","last_undegraded":"2024-01-11T21:44:21.491077+0000","last_fullsized":"2024-01-11T21:44:21.491077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.490886+0000","last_change":"2024-01-11T21:44:21.488108+0000","last_active":"2024-01-11T21:44:21.490886+0000","last_peered":"2024-01-11T21:44:21.490886+0000","last_clean":"2024-01-11T21:44:21.490886+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T21:44:21.490886+0000","last_undegraded":"2024-01-11T21:44:21.490886+0000","last_fullsized":"2024-01-11T21:44:21.490886+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491563+0000","last_change":"2024-01-11T21:44:21.489846+0000","last_active":"2024-01-11T21:44:21.491563+0000","last_peered":"2024-01-11T21:44:21.491563+0000","last_clean":"2024-01-11T21:44:21.491563+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T21:44:21.491563+0000","last_undegraded":"2024-01-11T21:44:21.491563+0000","last_fullsized":"2024-01-11T21:44:21.491563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.492390+0000","last_change":"2024-01-11T21:44:21.490369+0000","last_active":"2024-01-11T21:44:21.492390+0000","last_peered":"2024-01-11T21:44:21.492390+0000","last_clean":"2024-01-11T21:44:21.492390+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T21:44:21.492390+0000","last_undegraded":"2024-01-11T21:44:21.492390+0000","last_fullsized":"2024-01-11T21:44:21.492390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.489351+0000","last_change":"2024-01-11T21:44:21.489351+0000","last_active":"2024-01-11T21:44:21.489351+0000","last_peered":"2024-01-11T21:44:21.489351+0000","last_clean":"2024-01-11T21:44:21.489351+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T21:44:21.489351+0000","last_undegraded":"2024-01-11T21:44:21.489351+0000","last_fullsized":"2024-01-11T21:44:21.489351+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.490305+0000","last_change":"2024-01-11T21:44:21.490305+0000","last_active":"2024-01-11T21:44:21.490305+0000","last_peered":"2024-01-11T21:44:21.490305+0000","last_clean":"2024-01-11T21:44:21.490305+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-11T21:44:21.490305+0000","last_undegraded":"2024-01-11T21:44:21.490305+0000","last_fullsized":"2024-01-11T21:44:21.490305+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:33:17.886132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491004+0000","last_change":"2024-01-11T21:44:21.491004+0000","last_active":"2024-01-11T21:44:21.491004+0000","last_peered":"2024-01-11T21:44:21.491004+0000","last_clean":"2024-01-11T21:44:21.491004+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-11T21:44:21.491004+0000","last_undegraded":"2024-01-11T21:44:21.491004+0000","last_fullsized":"2024-01-11T21:44:21.491004+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:21.279733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491105+0000","last_change":"2024-01-11T21:44:21.488228+0000","last_active":"2024-01-11T21:44:21.491105+0000","last_peered":"2024-01-11T21:44:21.491105+0000","last_clean":"2024-01-11T21:44:21.491105+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T21:44:21.491105+0000","last_undegraded":"2024-01-11T21:44:21.491105+0000","last_fullsized":"2024-01-11T21:44:21.491105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.479456+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T21:44:21.479456+0000","last_peered":"2024-01-11T21:44:21.479456+0000","last_clean":"2024-01-11T21:44:21.479456+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T21:44:21.479456+0000","last_undegraded":"2024-01-11T21:44:21.479456+0000","last_fullsized":"2024-01-11T21:44:21.479456+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301496,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070344,"statfs":{"total":96636764160,"available":96328032256,"internally_reserved":0,"allocated":114688,"data_stored":31159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302084,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069756,"statfs":{"total":96636764160,"available":96327430144,"internally_reserved":0,"allocated":716800,"data_stored":624993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302072,"kb_used_data":688,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069768,"statfs":{"total":96636764160,"available":96327442432,"internally_reserved":0,"allocated":704512,"data_stored":620766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T21:44:24.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T21:44:24.862 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:44:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-11T21:44:24.875 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-01-11T21:44:24.618358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":905660,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":904128,"kb_avail":282209860,"statfs":{"total":289910292480,"available":288982896640,"internally_reserved":0,"allocated":1544192,"data_stored":1280384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":925827072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.148689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491077+0000","last_change":"2024-01-11T21:44:21.487885+0000","last_active":"2024-01-11T21:44:21.491077+0000","last_peered":"2024-01-11T21:44:21.491077+0000","last_clean":"2024-01-11T21:44:21.491077+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T21:44:21.491077+0000","last_undegraded":"2024-01-11T21:44:21.491077+0000","last_fullsized":"2024-01-11T21:44:21.491077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.490886+0000","last_change":"2024-01-11T21:44:21.488108+0000","last_active":"2024-01-11T21:44:21.490886+0000","last_peered":"2024-01-11T21:44:21.490886+0000","last_clean":"2024-01-11T21:44:21.490886+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T21:44:21.490886+0000","last_undegraded":"2024-01-11T21:44:21.490886+0000","last_fullsized":"2024-01-11T21:44:21.490886+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491563+0000","last_change":"2024-01-11T21:44:21.489846+0000","last_active":"2024-01-11T21:44:21.491563+0000","last_peered":"2024-01-11T21:44:21.491563+0000","last_clean":"2024-01-11T21:44:21.491563+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T21:44:21.491563+0000","last_undegraded":"2024-01-11T21:44:21.491563+0000","last_fullsized":"2024-01-11T21:44:21.491563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.492390+0000","last_change":"2024-01-11T21:44:21.490369+0000","last_active":"2024-01-11T21:44:21.492390+0000","last_peered":"2024-01-11T21:44:21.492390+0000","last_clean":"2024-01-11T21:44:21.492390+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T21:44:21.492390+0000","last_undegraded":"2024-01-11T21:44:21.492390+0000","last_fullsized":"2024-01-11T21:44:21.492390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491764+0000","last_change":"2024-01-11T21:44:21.489351+0000","last_active":"2024-01-11T21:44:21.491764+0000","last_peered":"2024-01-11T21:44:21.491764+0000","last_clean":"2024-01-11T21:44:21.491764+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T21:44:21.491764+0000","last_undegraded":"2024-01-11T21:44:21.491764+0000","last_fullsized":"2024-01-11T21:44:21.491764+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.492315+0000","last_change":"2024-01-11T21:44:21.490305+0000","last_active":"2024-01-11T21:44:21.492315+0000","last_peered":"2024-01-11T21:44:21.492315+0000","last_clean":"2024-01-11T21:44:21.492315+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-11T21:44:21.492315+0000","last_undegraded":"2024-01-11T21:44:21.492315+0000","last_fullsized":"2024-01-11T21:44:21.492315+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:33:17.886132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.492369+0000","last_change":"2024-01-11T21:44:21.491004+0000","last_active":"2024-01-11T21:44:21.492369+0000","last_peered":"2024-01-11T21:44:21.492369+0000","last_clean":"2024-01-11T21:44:21.492369+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-11T21:44:21.492369+0000","last_undegraded":"2024-01-11T21:44:21.492369+0000","last_fullsized":"2024-01-11T21:44:21.492369+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:21.279733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.491105+0000","last_change":"2024-01-11T21:44:21.488228+0000","last_active":"2024-01-11T21:44:21.491105+0000","last_peered":"2024-01-11T21:44:21.491105+0000","last_clean":"2024-01-11T21:44:21.491105+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T21:44:21.491105+0000","last_undegraded":"2024-01-11T21:44:21.491105+0000","last_fullsized":"2024-01-11T21:44:21.491105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T21:44:21.479456+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T21:44:21.479456+0000","last_peered":"2024-01-11T21:44:21.479456+0000","last_clean":"2024-01-11T21:44:21.479456+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T21:44:21.479456+0000","last_undegraded":"2024-01-11T21:44:21.479456+0000","last_fullsized":"2024-01-11T21:44:21.479456+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301496,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070344,"statfs":{"total":96636764160,"available":96328032256,"internally_reserved":0,"allocated":114688,"data_stored":31159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302084,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069756,"statfs":{"total":96636764160,"available":96327430144,"internally_reserved":0,"allocated":716800,"data_stored":624993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302080,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069760,"statfs":{"total":96636764160,"available":96327434240,"internally_reserved":0,"allocated":712704,"data_stored":624232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T21:44:24.876 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-11T21:44:24.876 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-11T21:44:24.877 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-11T21:44:24.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-11T21:44:25.181 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-11T21:44:25.181 INFO:teuthology.orchestra.run.smithi178.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-11T21:44:25.195 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-11T21:44:25.195 INFO:teuthology.run_tasks:Running task exec... 2024-01-11T21:44:25.208 INFO:teuthology.task.exec:Executing custom commands... 2024-01-11T21:44:25.209 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi178.front.sepia.ceph.com 2024-01-11T21:44:25.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2024-01-11T21:44:26.508 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'cache' created 2024-01-11T21:44:26.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add rbd cache' 2024-01-11T21:44:27.518 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'cache' is now (or already was) a tier of 'rbd' 2024-01-11T21:44:27.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2024-01-11T21:44:28.522 INFO:teuthology.orchestra.run.smithi178.stderr:set cache-mode for pool 'cache' to writeback 2024-01-11T21:44:28.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay rbd cache' 2024-01-11T21:44:29.526 INFO:teuthology.orchestra.run.smithi178.stderr:overlay for 'rbd' is now (or already was) 'cache' 2024-01-11T21:44:29.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2024-01-11T21:44:30.530 INFO:teuthology.orchestra.run.smithi178.stderr:set pool 3 hit_set_type to bloom 2024-01-11T21:44:30.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2024-01-11T21:44:31.535 INFO:teuthology.orchestra.run.smithi178.stderr:set pool 3 hit_set_count to 8 2024-01-11T21:44:31.549 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2024-01-11T21:44:32.537 INFO:teuthology.orchestra.run.smithi178.stderr:set pool 3 hit_set_period to 60 2024-01-11T21:44:32.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2024-01-11T21:44:33.545 INFO:teuthology.orchestra.run.smithi178.stderr:set pool 3 target_max_objects to 250 2024-01-11T21:44:33.560 INFO:teuthology.run_tasks:Running task workunit... 2024-01-11T21:44:33.572 INFO:tasks.workunit:Pulling workunits from ref 49908e15c7044e78475683aa6ccd77bd187be2f4 2024-01-11T21:44:33.573 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-11T21:44:33.573 DEBUG:teuthology.orchestra.run.smithi178:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-11T21:44:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T21:44:33.579 INFO:teuthology.orchestra.run.smithi178.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-11T21:44:33.579 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-01-11T21:44:33.629 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-01-11T21:44:33.630 DEBUG:teuthology.orchestra.run.smithi178:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-01-11T21:44:33.636 INFO:tasks.workunit:timeout=3h 2024-01-11T21:44:33.636 INFO:tasks.workunit:cleanup=True 2024-01-11T21:44:33.637 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 49908e15c7044e78475683aa6ccd77bd187be2f4 2024-01-11T21:44:33.645 INFO:tasks.workunit.client.0.smithi178.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-11T21:45:15.892 INFO:tasks.workunit.client.0.smithi178.stderr:Note: switching to '49908e15c7044e78475683aa6ccd77bd187be2f4'. 2024-01-11T21:45:15.892 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-01-11T21:45:15.892 INFO:tasks.workunit.client.0.smithi178.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-11T21:45:15.892 INFO:tasks.workunit.client.0.smithi178.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-11T21:45:15.892 INFO:tasks.workunit.client.0.smithi178.stderr:state without impacting any branches by switching back to a branch. 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr: git switch -c 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr:Or undo this operation with: 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr: git switch - 2024-01-11T21:45:15.893 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-01-11T21:45:15.894 INFO:tasks.workunit.client.0.smithi178.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-11T21:45:15.894 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-01-11T21:45:15.894 INFO:tasks.workunit.client.0.smithi178.stderr:HEAD is now at 49908e15c70 qa/suites: install pytest for pybind tasks 2024-01-11T21:45:15.896 DEBUG:teuthology.orchestra.run.smithi178:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-11T21:45:15.952 INFO:tasks.workunit.client.0.smithi178.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-11T21:45:15.956 INFO:tasks.workunit.client.0.smithi178.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-11T21:45:15.956 INFO:tasks.workunit.client.0.smithi178.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-11T21:45:16.767 INFO:tasks.workunit.client.0.smithi178.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-11T21:45:16.877 INFO:tasks.workunit.client.0.smithi178.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-11T21:45:16.933 INFO:tasks.workunit.client.0.smithi178.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-11T21:45:16.935 INFO:tasks.workunit.client.0.smithi178.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-11T21:45:16.935 INFO:tasks.workunit.client.0.smithi178.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-11T21:45:16.996 INFO:tasks.workunit.client.0.smithi178.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-11T21:45:17.001 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-11T21:45:17.001 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-11T21:45:17.006 INFO:tasks.workunit:Running workunits matching rbd/cli_generic.sh on client.0... 2024-01-11T21:45:17.007 INFO:tasks.workunit:Running workunit rbd/cli_generic.sh... 2024-01-11T21:45:17.008 DEBUG:teuthology.orchestra.run.smithi178:workunit test rbd/cli_generic.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=49908e15c7044e78475683aa6ccd77bd187be2f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh 2024-01-11T21:45:17.020 INFO:tasks.workunit.client.0.smithi178.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh 2024-01-11T21:45:17.021 INFO:tasks.workunit.client.0.smithi178.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-01-11T21:45:17.021 INFO:tasks.workunit.client.0.smithi178.stderr:++ TIMEOUT=300 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ MAX_TIMEOUT=15 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ PG_NUM=4 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ TMPDIR=/tmp 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ type xmlstarlet 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:++ XMLSTARLET=xmlstarlet 2024-01-11T21:45:17.022 INFO:tasks.workunit.client.0.smithi178.stderr:+++ uname 2024-01-11T21:45:17.023 INFO:tasks.workunit.client.0.smithi178.stderr:++ '[' Linux = FreeBSD ']' 2024-01-11T21:45:17.023 INFO:tasks.workunit.client.0.smithi178.stderr:++ SED=sed 2024-01-11T21:45:17.023 INFO:tasks.workunit.client.0.smithi178.stderr:++ AWK=awk 2024-01-11T21:45:17.024 INFO:tasks.workunit.client.0.smithi178.stderr:+++ stty -a 2024-01-11T21:45:17.024 INFO:tasks.workunit.client.0.smithi178.stderr:+++ head -1 2024-01-11T21:45:17.024 INFO:tasks.workunit.client.0.smithi178.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-01-11T21:45:17.030 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-01-11T21:45:17.031 INFO:tasks.workunit.client.0.smithi178.stderr:++ termwidth= 2024-01-11T21:45:17.031 INFO:tasks.workunit.client.0.smithi178.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-01-11T21:45:17.032 INFO:tasks.workunit.client.0.smithi178.stderr:++ DIFFCOLOPTS='-y ' 2024-01-11T21:45:17.032 INFO:tasks.workunit.client.0.smithi178.stderr:++ KERNCORE=kernel.core_pattern 2024-01-11T21:45:17.032 INFO:tasks.workunit.client.0.smithi178.stderr:++ EXTRA_OPTS= 2024-01-11T21:45:17.042 INFO:tasks.workunit.client.0.smithi178.stderr:++ test '' = TESTS 2024-01-11T21:45:17.042 INFO:tasks.workunit.client.0.smithi178.stderr:+ export RBD_FORCE_ALLOW_V1=1 2024-01-11T21:45:17.042 INFO:tasks.workunit.client.0.smithi178.stderr:+ RBD_FORCE_ALLOW_V1=1 2024-01-11T21:45:17.042 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:17.042 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:17.043 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v '^0$' 2024-01-11T21:45:17.090 INFO:tasks.workunit.client.0.smithi178.stderr:+ IMGS='testimg1 testimg2 testimg3 testimg4 testimg5 testimg6 testimg-diff1 testimg-diff2 testimg-diff3 foo foo2 bar bar2 test1 test2 test3 test4 clone2' 2024-01-11T21:45:17.090 INFO:tasks.workunit.client.0.smithi178.stderr:+ tiered=0 2024-01-11T21:45:17.090 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd dump 2024-01-11T21:45:17.091 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^pool' 2024-01-11T21:45:17.091 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ''\''rbd'\''' 2024-01-11T21:45:17.091 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep tier 2024-01-11T21:45:17.472 INFO:tasks.workunit.client.0.smithi178.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 lfor 18/18/18 flags hashpspool tiers 3 read_tier 3 write_tier 3 stripe_width 0 application rbd 2024-01-11T21:45:17.473 INFO:tasks.workunit.client.0.smithi178.stderr:+ tiered=1 2024-01-11T21:45:17.486 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_pool_image_args 2024-01-11T21:45:17.486 INFO:tasks.workunit.client.0.smithi178.stdout:testing pool and image args... 2024-01-11T21:45:17.486 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing pool and image args...' 2024-01-11T21:45:17.487 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:45:17.487 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:17.601 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:17.714 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:17.833 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:17.940 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.051 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.172 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.289 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.403 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.519 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.758 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.875 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:18.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:19.113 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:19.225 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:19.335 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:19.456 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:19.578 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool delete test test --yes-i-really-really-mean-it 2024-01-11T21:45:19.958 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'test' does not exist 2024-01-11T21:45:19.966 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool create test 32 2024-01-11T21:45:20.633 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'test' already exists 2024-01-11T21:45:20.641 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd pool init test 2024-01-11T21:45:23.558 INFO:tasks.workunit.client.0.smithi178.stderr:+ truncate -s 1 /tmp/empty /tmp/empty@snap 2024-01-11T21:45:23.572 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:23.572 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:45:23.572 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:23.621 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:45:23.621 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 1 test1 2024-01-11T21:45:23.655 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:23.649+0000 7f125ce0a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:25.580 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:25.580 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test1 2024-01-11T21:45:25.618 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import --image test2 /tmp/empty 2024-01-11T21:45:25.862 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:25.857+0000 7f2066a354c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:25.883 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:25.884 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: --image is deprecated, use --dest 2024-01-11T21:45:25.888 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:25.889 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test2 2024-01-11T21:45:25.931 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --dest test3 import /tmp/empty 2024-01-11T21:45:25.973 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:25.969+0000 7f3f18b704c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:25.991 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test3 2024-01-11T21:45:26.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import /tmp/empty foo 2024-01-11T21:45:26.076 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:26.073+0000 7f5f462824c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:26.093 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:26.100 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:26.100 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q foo 2024-01-11T21:45:26.142 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import --dest test/empty@snap /tmp/empty 2024-01-11T21:45:26.170 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-01-11T21:45:26.171 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:45:26.171 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import /tmp/empty test/empty@snap 2024-01-11T21:45:26.198 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-01-11T21:45:26.200 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:45:26.200 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import --image test/empty@snap /tmp/empty 2024-01-11T21:45:26.229 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-01-11T21:45:26.229 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: --image is deprecated, use --dest 2024-01-11T21:45:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:45:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import /tmp/empty@snap 2024-01-11T21:45:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-01-11T21:45:26.260 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:45:26.260 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:26.261 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:26.261 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:45:26.299 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:45:26.300 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import /tmp/empty test/test1 2024-01-11T21:45:26.339 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:26.333+0000 7f56fa46f4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:27.598 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:27.601 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:27.601 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test1 2024-01-11T21:45:27.638 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd -p test import /tmp/empty test2 2024-01-11T21:45:27.880 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:27.877+0000 7f45ec4f64c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:27.903 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:27.904 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-01-11T21:45:27.908 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:27.909 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test2 2024-01-11T21:45:27.953 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --image test3 -p test import /tmp/empty 2024-01-11T21:45:27.995 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:27.989+0000 7feed46ee4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.014 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.016 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: --image is deprecated, use --dest 2024-01-11T21:45:28.016 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-01-11T21:45:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test3 2024-01-11T21:45:28.063 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --image test4 -p test import /tmp/empty 2024-01-11T21:45:28.108 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:28.105+0000 7f83239094c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.126 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.127 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: --image is deprecated, use --dest 2024-01-11T21:45:28.127 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-01-11T21:45:28.132 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.132 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test4 2024-01-11T21:45:28.176 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --dest test5 -p test import /tmp/empty 2024-01-11T21:45:28.217 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:28.213+0000 7fdb4f43a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.237 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.238 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-01-11T21:45:28.242 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.242 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test5 2024-01-11T21:45:28.287 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --dest test6 --dest-pool test import /tmp/empty 2024-01-11T21:45:28.327 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:28.321+0000 7f43f38a14c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.346 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.352 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.352 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test6 2024-01-11T21:45:28.399 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --image test7 --dest-pool test import /tmp/empty 2024-01-11T21:45:28.441 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:28.437+0000 7fab7bb934c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.462 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.463 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: --image is deprecated, use --dest 2024-01-11T21:45:28.468 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.468 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test7 2024-01-11T21:45:28.511 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --image test/test8 import /tmp/empty 2024-01-11T21:45:28.553 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:28.549+0000 7fcfd9b124c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.571 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.572 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: --image is deprecated, use --dest 2024-01-11T21:45:28.577 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.577 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test8 2024-01-11T21:45:28.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --dest test/test9 import /tmp/empty 2024-01-11T21:45:28.658 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:28.653+0000 7f9738a924c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.678 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.684 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.684 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test9 2024-01-11T21:45:28.729 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import --pool test /tmp/empty 2024-01-11T21:45:28.769 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:28.765+0000 7fc5e70bf4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:28.788 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:45:28.789 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-01-11T21:45:28.793 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:28.793 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q empty 2024-01-11T21:45:28.838 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy test/test9 test10 2024-01-11T21:45:30.527 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:45:30.533 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:30.533 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -qv test10 2024-01-11T21:45:30.573 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:30.573 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test10 2024-01-11T21:45:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy test/test9 test/test10 2024-01-11T21:45:30.692 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:45:30.699 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:30.699 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test10 2024-01-11T21:45:30.742 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy --pool test test10 --dest-pool test test11 2024-01-11T21:45:30.834 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:45:30.840 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:30.840 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -q test11 2024-01-11T21:45:30.888 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy --dest-pool rbd --pool test test11 test12 2024-01-11T21:45:30.978 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:45:30.984 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:30.984 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test12 2024-01-11T21:45:31.029 INFO:tasks.workunit.client.0.smithi178.stdout:test12 2024-01-11T21:45:31.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls test 2024-01-11T21:45:31.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -qv test12 2024-01-11T21:45:31.073 INFO:tasks.workunit.client.0.smithi178.stderr:+ rm -f /tmp/empty /tmp/empty@snap 2024-01-11T21:45:31.074 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool delete test test --yes-i-really-really-mean-it 2024-01-11T21:45:31.612 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'test' does not exist 2024-01-11T21:45:31.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-01-11T21:45:31.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm foo 2024-01-11T21:45:31.694 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:31.697 INFO:tasks.workunit.client.0.smithi178.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-01-11T21:45:31.697 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:45:31.748 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:31.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-01-11T21:45:31.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test10 2024-01-11T21:45:31.876 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:31.880 INFO:tasks.workunit.client.0.smithi178.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-01-11T21:45:31.880 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test12 2024-01-11T21:45:32.009 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:32.014 INFO:tasks.workunit.client.0.smithi178.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-01-11T21:45:32.014 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:45:32.079 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:32.084 INFO:tasks.workunit.client.0.smithi178.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-01-11T21:45:32.084 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test3 2024-01-11T21:45:32.147 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:32.151 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_rename 2024-01-11T21:45:32.151 INFO:tasks.workunit.client.0.smithi178.stdout:testing rename... 2024-01-11T21:45:32.152 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing rename...' 2024-01-11T21:45:32.152 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:45:32.152 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:32.267 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:32.385 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:32.502 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:32.823 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:32.930 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.036 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.150 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.264 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.381 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.495 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.606 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.842 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:33.956 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:34.073 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:34.190 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:34.302 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:34.418 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 1 -s 1 foo 2024-01-11T21:45:34.447 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:45:34.459 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:34.457+0000 7f12340784c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:34.469 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 bar 2024-01-11T21:45:34.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename foo foo2 2024-01-11T21:45:34.606 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename foo2 bar 2024-01-11T21:45:34.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep exists 2024-01-11T21:45:34.666 INFO:tasks.workunit.client.0.smithi178.stdout:2024-01-11T21:45:34.653+0000 7f035b6d84c0 -1 librbd::Operations: rbd image bar already exists 2024-01-11T21:45:34.666 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: rename error: (17) File exists 2024-01-11T21:45:34.667 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename bar bar2 2024-01-11T21:45:34.754 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename bar2 foo2 2024-01-11T21:45:34.755 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep exists 2024-01-11T21:45:34.816 INFO:tasks.workunit.client.0.smithi178.stdout:2024-01-11T21:45:34.805+0000 7f2dcae654c0 -1 librbd::Operations: rbd image foo2 already exists 2024-01-11T21:45:34.816 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: rename error: (17) File exists 2024-01-11T21:45:34.816 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool create rbd2 8 2024-01-11T21:45:35.625 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd2' already exists 2024-01-11T21:45:35.634 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd pool init rbd2 2024-01-11T21:45:38.545 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -p rbd2 -s 1 foo 2024-01-11T21:45:38.580 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:38.577+0000 7fcf78dde4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:40.510 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename rbd2/foo rbd2/bar 2024-01-11T21:45:40.578 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd -p rbd2 ls 2024-01-11T21:45:40.578 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep bar 2024-01-11T21:45:40.623 INFO:tasks.workunit.client.0.smithi178.stdout:bar 2024-01-11T21:45:40.623 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename rbd2/bar foo 2024-01-11T21:45:40.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename --pool rbd2 foo bar 2024-01-11T21:45:40.766 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename rbd2/bar --dest-pool rbd foo 2024-01-11T21:45:40.795 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: mv/rename across pools not supported 2024-01-11T21:45:40.795 INFO:tasks.workunit.client.0.smithi178.stderr:source pool: rbd2 dest pool: rbd 2024-01-11T21:45:40.796 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rename --pool rbd2 bar --dest-pool rbd2 foo 2024-01-11T21:45:40.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd -p rbd2 ls 2024-01-11T21:45:40.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep foo 2024-01-11T21:45:40.911 INFO:tasks.workunit.client.0.smithi178.stdout:foo 2024-01-11T21:45:40.912 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-01-11T21:45:41.610 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd2' does not exist 2024-01-11T21:45:41.617 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:45:41.618 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:41.730 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:41.845 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:41.961 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.073 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.193 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.307 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.425 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.541 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.767 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:42.889 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.002 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.228 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.340 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.452 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.570 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.686 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.802 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_ls 2024-01-11T21:45:43.838 INFO:tasks.workunit.client.0.smithi178.stdout:testing ls... 2024-01-11T21:45:43.839 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing ls...' 2024-01-11T21:45:43.839 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:45:43.839 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:43.918 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.032 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.141 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.265 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.383 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.505 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.622 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.740 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.861 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:44.979 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.096 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.216 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.332 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.448 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.559 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.790 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:45.906 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 1 -s 1 test1 2024-01-11T21:45:45.931 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:45:45.943 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:45.941+0000 7f99349494c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:45.956 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 1 -s 1 test2 2024-01-11T21:45:45.984 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:45:45.995 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:45.993+0000 7f75302954c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:46.006 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:46.006 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:45:46.052 INFO:tasks.workunit.client.0.smithi178.stdout:test1 2024-01-11T21:45:46.052 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:46.052 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test2 2024-01-11T21:45:46.090 INFO:tasks.workunit.client.0.smithi178.stdout:test2 2024-01-11T21:45:46.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:46.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:46.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:45:46.132 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:45:46.132 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:45:46.133 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test1.*1 MiB.*1' 2024-01-11T21:45:46.186 INFO:tasks.workunit.client.0.smithi178.stdout:test1 1 MiB 1 2024-01-11T21:45:46.187 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:45:46.187 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test2.*1 MiB.*1' 2024-01-11T21:45:46.240 INFO:tasks.workunit.client.0.smithi178.stdout:test2 1 MiB 1 2024-01-11T21:45:46.240 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:45:46.302 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:46.306 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:45:46.367 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:46.372 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test1 2024-01-11T21:45:46.442 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test2 2024-01-11T21:45:46.508 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:46.508 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:45:46.544 INFO:tasks.workunit.client.0.smithi178.stdout:test1 2024-01-11T21:45:46.545 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:46.545 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test2 2024-01-11T21:45:46.585 INFO:tasks.workunit.client.0.smithi178.stdout:test2 2024-01-11T21:45:46.586 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:46.586 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:46.586 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:45:46.625 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:45:46.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:45:46.626 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test1.*1 MiB.*2' 2024-01-11T21:45:46.680 INFO:tasks.workunit.client.0.smithi178.stdout:test1 1 MiB 2 2024-01-11T21:45:46.681 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:45:46.681 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test2.*1 MiB.*2' 2024-01-11T21:45:46.738 INFO:tasks.workunit.client.0.smithi178.stdout:test2 1 MiB 2 2024-01-11T21:45:46.738 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:45:46.885 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:46.881+0000 7f37eec4c700 0 -- 172.21.15.178:0/432849720 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f37d4008740 msgr2=0x7f37d400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:45:46.897 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:46.901 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:45:47.058 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:47.053+0000 7f445aa58700 0 -- 172.21.15.178:0/1249931830 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f4440036c40 msgr2=0x7f4440039020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:45:47.067 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:47.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test1 2024-01-11T21:45:47.137 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 1 -s 1 test2 2024-01-11T21:45:47.166 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:45:47.176 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:47.173+0000 7f6efd6294c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:45:47.229 INFO:tasks.workunit.client.0.smithi178.stdout:test1 2024-01-11T21:45:47.229 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test2 2024-01-11T21:45:47.272 INFO:tasks.workunit.client.0.smithi178.stdout:test2 2024-01-11T21:45:47.272 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:47.273 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:47.273 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:45:47.316 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:45:47.317 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:45:47.317 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test1.*1 MiB.*2' 2024-01-11T21:45:47.375 INFO:tasks.workunit.client.0.smithi178.stdout:test1 1 MiB 2 2024-01-11T21:45:47.376 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:45:47.376 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test2.*1 MiB.*1' 2024-01-11T21:45:47.438 INFO:tasks.workunit.client.0.smithi178.stdout:test2 1 MiB 1 2024-01-11T21:45:47.438 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:45:47.438 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:47.554 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:47.878 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:47.991 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.108 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.228 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.341 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.459 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.577 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.693 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.814 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:48.921 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:49.031 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:49.147 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:49.383 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:49.506 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:49.616 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:49.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:45:50.055 INFO:tasks.workunit.client.0.smithi178.stderr:++ seq -w 00 99 2024-01-11T21:45:50.255 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.255 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.00 -s 1 2024-01-11T21:45:50.302 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.297+0000 7f57c8ede4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.312 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.312 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.01 -s 1 2024-01-11T21:45:50.354 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.349+0000 7fbe362834c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.362 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.363 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.02 -s 1 2024-01-11T21:45:50.404 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.401+0000 7fedb44ee4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.415 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.415 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.03 -s 1 2024-01-11T21:45:50.454 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.449+0000 7fc9ff7f54c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.464 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.464 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.04 -s 1 2024-01-11T21:45:50.507 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.501+0000 7f688bad24c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.517 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.517 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.05 -s 1 2024-01-11T21:45:50.556 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.553+0000 7f84e90844c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.566 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.566 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.06 -s 1 2024-01-11T21:45:50.606 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.601+0000 7fb9eb3f54c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.615 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.615 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.07 -s 1 2024-01-11T21:45:50.654 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.649+0000 7fa240f1e4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.663 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.663 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.08 -s 1 2024-01-11T21:45:50.704 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.701+0000 7fd6186f84c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.713 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.713 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.09 -s 1 2024-01-11T21:45:50.752 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.749+0000 7f233e4424c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.10 -s 1 2024-01-11T21:45:50.799 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.793+0000 7f45c96794c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.809 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.809 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.11 -s 1 2024-01-11T21:45:50.848 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.845+0000 7f1b54f2d4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.858 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.858 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.12 -s 1 2024-01-11T21:45:50.899 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.893+0000 7efd8986a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.909 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.909 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.13 -s 1 2024-01-11T21:45:50.950 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.945+0000 7fa7e55b04c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:50.959 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:50.959 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.14 -s 1 2024-01-11T21:45:50.994 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:50.989+0000 7fcef76f64c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.003 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.003 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.15 -s 1 2024-01-11T21:45:51.042 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.037+0000 7f3186ac24c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.052 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.052 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.16 -s 1 2024-01-11T21:45:51.091 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.085+0000 7f8bbb62a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.102 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.102 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.17 -s 1 2024-01-11T21:45:51.140 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.137+0000 7fcfd72fa4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.150 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.150 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.18 -s 1 2024-01-11T21:45:51.192 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.189+0000 7f9225bcc4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.203 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.203 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.19 -s 1 2024-01-11T21:45:51.244 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.241+0000 7f3979fa34c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.253 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.253 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.20 -s 1 2024-01-11T21:45:51.290 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.285+0000 7ff81082b4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.299 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.299 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.21 -s 1 2024-01-11T21:45:51.340 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.337+0000 7fdcde8884c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.350 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.350 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.22 -s 1 2024-01-11T21:45:51.392 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.389+0000 7f2678b314c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.403 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.403 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.23 -s 1 2024-01-11T21:45:51.441 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.437+0000 7f3ac002a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.450 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.450 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.24 -s 1 2024-01-11T21:45:51.488 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.485+0000 7f164b6354c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.499 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.499 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.25 -s 1 2024-01-11T21:45:51.537 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.533+0000 7fe743b364c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.547 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.547 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.26 -s 1 2024-01-11T21:45:51.586 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.581+0000 7f3936fc94c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.597 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.597 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.27 -s 1 2024-01-11T21:45:51.636 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.633+0000 7f60f11cd4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.645 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.645 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.28 -s 1 2024-01-11T21:45:51.685 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.681+0000 7feba3ea14c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.696 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.696 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.29 -s 1 2024-01-11T21:45:51.734 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.729+0000 7f9588a634c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.743 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.743 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.30 -s 1 2024-01-11T21:45:51.782 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.777+0000 7fde92ea44c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.31 -s 1 2024-01-11T21:45:51.832 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.829+0000 7fdd2e0974c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.842 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.842 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.32 -s 1 2024-01-11T21:45:51.883 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.877+0000 7fe786cc74c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.892 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.893 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.33 -s 1 2024-01-11T21:45:51.934 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.929+0000 7f6933eba4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.943 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.943 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.34 -s 1 2024-01-11T21:45:51.983 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:51.977+0000 7fad601ca4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:51.993 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:51.993 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.35 -s 1 2024-01-11T21:45:52.034 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.029+0000 7f1c09e084c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.043 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.043 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.36 -s 1 2024-01-11T21:45:52.082 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.077+0000 7f3fd3b684c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.37 -s 1 2024-01-11T21:45:52.131 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.129+0000 7f3cdd4e34c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.140 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.140 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.38 -s 1 2024-01-11T21:45:52.177 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.173+0000 7f1ec42e84c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.39 -s 1 2024-01-11T21:45:52.223 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.217+0000 7fa2ceb844c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.232 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.233 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.40 -s 1 2024-01-11T21:45:52.270 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.265+0000 7f1e042754c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.280 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.280 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.41 -s 1 2024-01-11T21:45:52.319 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.313+0000 7ff7ba42b4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.328 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.328 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.42 -s 1 2024-01-11T21:45:52.368 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.365+0000 7f5e078554c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.378 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.378 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.43 -s 1 2024-01-11T21:45:52.417 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.413+0000 7f1e20d7f4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.426 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.426 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.44 -s 1 2024-01-11T21:45:52.461 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.457+0000 7f011589a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.471 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.471 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.45 -s 1 2024-01-11T21:45:52.512 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.509+0000 7f4a8b0a34c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.522 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.522 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.46 -s 1 2024-01-11T21:45:52.562 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.557+0000 7f85354854c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.572 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.572 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.47 -s 1 2024-01-11T21:45:52.613 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.609+0000 7f7fb65e44c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.623 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.623 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.48 -s 1 2024-01-11T21:45:52.664 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.661+0000 7f2de336f4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.674 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.674 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.49 -s 1 2024-01-11T21:45:52.716 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.713+0000 7f2886b1b4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.728 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.728 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.50 -s 1 2024-01-11T21:45:52.766 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.761+0000 7f075773b4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.777 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.777 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.51 -s 1 2024-01-11T21:45:52.813 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.809+0000 7fbe727de4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.823 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.823 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.52 -s 1 2024-01-11T21:45:52.865 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.861+0000 7fafbbee44c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.876 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.876 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.53 -s 1 2024-01-11T21:45:52.916 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.913+0000 7f1c634844c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.927 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.927 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.54 -s 1 2024-01-11T21:45:52.967 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:52.961+0000 7f1339a1b4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:52.978 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:52.978 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.55 -s 1 2024-01-11T21:45:53.014 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.009+0000 7fa3142714c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.56 -s 1 2024-01-11T21:45:53.058 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.053+0000 7fbf1a3d94c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.067 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.068 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.57 -s 1 2024-01-11T21:45:53.106 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.101+0000 7fbdba2214c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.115 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.115 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.58 -s 1 2024-01-11T21:45:53.153 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.149+0000 7f124c3744c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.163 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.163 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.59 -s 1 2024-01-11T21:45:53.203 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.197+0000 7f24e0c634c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.215 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.215 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.60 -s 1 2024-01-11T21:45:53.253 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.249+0000 7f42033d64c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.61 -s 1 2024-01-11T21:45:53.306 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.301+0000 7fbc4297d4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.316 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.316 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.62 -s 1 2024-01-11T21:45:53.352 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.349+0000 7f850fd024c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.362 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.362 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.63 -s 1 2024-01-11T21:45:53.399 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.393+0000 7fd0c99804c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.411 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.411 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.64 -s 1 2024-01-11T21:45:53.448 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.445+0000 7f28b2ee04c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.458 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.458 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.65 -s 1 2024-01-11T21:45:53.501 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.497+0000 7f0a3ac664c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.511 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.511 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.66 -s 1 2024-01-11T21:45:53.548 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.545+0000 7f2c8be404c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.559 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.559 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.67 -s 1 2024-01-11T21:45:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.593+0000 7fa86c66c4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.608 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.608 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.68 -s 1 2024-01-11T21:45:53.647 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.641+0000 7fa4912e34c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.657 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.658 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.69 -s 1 2024-01-11T21:45:53.690 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.685+0000 7f049c0114c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.701 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.701 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.70 -s 1 2024-01-11T21:45:53.741 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.737+0000 7f57828474c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.71 -s 1 2024-01-11T21:45:53.789 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.785+0000 7f8b758424c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.802 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.802 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.72 -s 1 2024-01-11T21:45:53.840 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.837+0000 7fe4087a94c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.854 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.854 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.73 -s 1 2024-01-11T21:45:53.890 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.885+0000 7facc3d4c4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.901 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.901 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.74 -s 1 2024-01-11T21:45:53.935 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.929+0000 7f4d9f6a44c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.946 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.75 -s 1 2024-01-11T21:45:53.986 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:53.981+0000 7f0193c944c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:53.996 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:53.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.76 -s 1 2024-01-11T21:45:54.038 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.033+0000 7fb665db74c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.048 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.048 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.77 -s 1 2024-01-11T21:45:54.085 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.081+0000 7f77b17184c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.095 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.095 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.78 -s 1 2024-01-11T21:45:54.133 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.129+0000 7fc7e3fe24c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.144 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.145 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.79 -s 1 2024-01-11T21:45:54.183 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.177+0000 7f87878a24c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.80 -s 1 2024-01-11T21:45:54.234 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.229+0000 7f9c013ee4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.243 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.243 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.81 -s 1 2024-01-11T21:45:54.279 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.277+0000 7fc7123d54c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.82 -s 1 2024-01-11T21:45:54.325 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.321+0000 7f41f16de4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.336 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.336 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.83 -s 1 2024-01-11T21:45:54.375 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.373+0000 7f1ab16324c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.387 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.387 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.84 -s 1 2024-01-11T21:45:54.423 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.417+0000 7fb7eca324c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.436 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.436 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.85 -s 1 2024-01-11T21:45:54.475 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.469+0000 7fbc1a8fd4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.488 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.488 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.86 -s 1 2024-01-11T21:45:54.529 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.525+0000 7f1eb42a94c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.87 -s 1 2024-01-11T21:45:54.574 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.569+0000 7fb3e35674c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.584 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.584 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.88 -s 1 2024-01-11T21:45:54.624 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.621+0000 7fcf518b34c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.636 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.89 -s 1 2024-01-11T21:45:54.673 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.669+0000 7f32dbeac4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.684 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.684 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.90 -s 1 2024-01-11T21:45:54.926 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.921+0000 7fe2b86364c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.941 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.941 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.91 -s 1 2024-01-11T21:45:54.979 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:54.973+0000 7f2b57c254c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:54.991 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:54.991 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.92 -s 1 2024-01-11T21:45:55.030 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.025+0000 7f55325e94c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.042 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.043 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.93 -s 1 2024-01-11T21:45:55.078 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.073+0000 7fbe1d6644c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.088 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.088 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.94 -s 1 2024-01-11T21:45:55.127 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.125+0000 7f2a4313a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.140 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.140 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.95 -s 1 2024-01-11T21:45:55.181 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.177+0000 7f2a67d1e4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.193 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.193 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.96 -s 1 2024-01-11T21:45:55.433 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.429+0000 7f90e880c4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.446 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.446 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.97 -s 1 2024-01-11T21:45:55.485 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.481+0000 7fd09895c4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.496 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.497 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.98 -s 1 2024-01-11T21:45:55.534 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.529+0000 7eff83d5a4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.543 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.543 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.99 -s 1 2024-01-11T21:45:55.583 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.577+0000 7f4bff2144c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:45:55.594 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:45:55.594 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:55.594 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 100 2024-01-11T21:45:55.636 INFO:tasks.workunit.client.0.smithi178.stdout:100 2024-01-11T21:45:55.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:45:55.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep image 2024-01-11T21:45:55.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:45:55.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 100 2024-01-11T21:45:55.727 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.721+0000 7f426c2f9700 0 -- 172.21.15.178:0/1900957466 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5619c25ac2d0 msgr2=0x5619c25a0220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:45:55.739 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:55.737+0000 7f426baf8700 0 -- 172.21.15.178:0/1900957466 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5619c264da20 msgr2=0x5619c2653c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:45:55.938 INFO:tasks.workunit.client.0.smithi178.stdout:100 2024-01-11T21:45:55.938 INFO:tasks.workunit.client.0.smithi178.stderr:++ seq -w 00 99 2024-01-11T21:45:55.939 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.939 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.00 2024-01-11T21:45:55.993 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:55.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:55.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.01 2024-01-11T21:45:56.054 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.058 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.058 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.02 2024-01-11T21:45:56.119 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.03 2024-01-11T21:45:56.188 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.193 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.193 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.04 2024-01-11T21:45:56.251 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.256 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.256 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.05 2024-01-11T21:45:56.319 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.322 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.322 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.06 2024-01-11T21:45:56.383 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.388 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.388 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.07 2024-01-11T21:45:56.447 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.452 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.452 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.08 2024-01-11T21:45:56.518 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.522 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.522 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.09 2024-01-11T21:45:56.585 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.590 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.590 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.10 2024-01-11T21:45:56.654 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.658 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.658 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.11 2024-01-11T21:45:56.720 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.12 2024-01-11T21:45:56.785 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.790 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.790 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.13 2024-01-11T21:45:56.848 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.854 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.854 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.14 2024-01-11T21:45:56.917 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.921 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.922 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.15 2024-01-11T21:45:56.983 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:56.987 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:56.987 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.16 2024-01-11T21:45:57.046 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.051 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.051 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.17 2024-01-11T21:45:57.115 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.119 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.119 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.18 2024-01-11T21:45:57.182 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.187 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.19 2024-01-11T21:45:57.251 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.256 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.256 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.20 2024-01-11T21:45:57.321 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.326 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.326 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.21 2024-01-11T21:45:57.388 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.392 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.393 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.22 2024-01-11T21:45:57.627 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:45:57.625+0000 7eff93639700 0 --2- 172.21.15.178:0/1045663357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56395a9a5650 0x56395aa44470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:45:57.666 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.669 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.670 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.23 2024-01-11T21:45:57.730 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.735 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.735 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.24 2024-01-11T21:45:57.797 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.801 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.802 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.25 2024-01-11T21:45:57.862 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.866 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.866 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.26 2024-01-11T21:45:57.929 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.933 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.933 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.27 2024-01-11T21:45:57.995 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:57.999 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:57.999 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.28 2024-01-11T21:45:58.056 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.060 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.060 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.29 2024-01-11T21:45:58.120 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.30 2024-01-11T21:45:58.189 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.194 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.194 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.31 2024-01-11T21:45:58.254 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.258 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.259 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.32 2024-01-11T21:45:58.320 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.325 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.325 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.33 2024-01-11T21:45:58.387 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.391 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.391 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.34 2024-01-11T21:45:58.456 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.460 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.460 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.35 2024-01-11T21:45:58.522 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.526 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.526 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.36 2024-01-11T21:45:58.587 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.591 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.591 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.37 2024-01-11T21:45:58.652 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.657 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.657 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.38 2024-01-11T21:45:58.721 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.726 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.726 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.39 2024-01-11T21:45:58.789 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.794 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.794 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.40 2024-01-11T21:45:58.857 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.862 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.862 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.41 2024-01-11T21:45:58.926 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.931 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.931 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.42 2024-01-11T21:45:58.993 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:58.998 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:58.998 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.43 2024-01-11T21:45:59.059 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.064 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.064 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.44 2024-01-11T21:45:59.124 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.129 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.129 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.45 2024-01-11T21:45:59.193 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.198 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.46 2024-01-11T21:45:59.259 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.264 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.47 2024-01-11T21:45:59.325 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.330 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.330 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.48 2024-01-11T21:45:59.393 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.397 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.397 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.49 2024-01-11T21:45:59.462 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.467 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.467 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.50 2024-01-11T21:45:59.526 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.529 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.529 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.51 2024-01-11T21:45:59.581 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.586 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.586 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.52 2024-01-11T21:45:59.649 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.53 2024-01-11T21:45:59.711 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.716 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.716 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.54 2024-01-11T21:45:59.775 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.779 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.779 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.55 2024-01-11T21:45:59.845 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.850 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.850 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.56 2024-01-11T21:45:59.914 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.919 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.919 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.57 2024-01-11T21:45:59.984 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:45:59.989 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:45:59.989 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.58 2024-01-11T21:46:00.045 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.050 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.050 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.59 2024-01-11T21:46:00.106 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.110 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.111 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.60 2024-01-11T21:46:00.167 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.171 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.171 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.61 2024-01-11T21:46:00.227 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.231 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.231 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.62 2024-01-11T21:46:00.283 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.287 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.287 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.63 2024-01-11T21:46:00.351 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.353 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.353 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.64 2024-01-11T21:46:00.412 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.416 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.416 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.65 2024-01-11T21:46:00.484 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.488 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.489 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.66 2024-01-11T21:46:00.553 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.558 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.558 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.67 2024-01-11T21:46:00.623 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.68 2024-01-11T21:46:00.693 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.697 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.697 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.69 2024-01-11T21:46:00.759 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.763 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.763 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.70 2024-01-11T21:46:00.821 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.826 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.826 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.71 2024-01-11T21:46:00.892 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.896 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.896 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.72 2024-01-11T21:46:00.954 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:00.959 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:00.959 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.73 2024-01-11T21:46:01.025 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.74 2024-01-11T21:46:01.085 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.090 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.090 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.75 2024-01-11T21:46:01.152 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.157 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.157 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.76 2024-01-11T21:46:01.424 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.429 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.429 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.77 2024-01-11T21:46:01.491 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.495 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.495 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.78 2024-01-11T21:46:01.557 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.561 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.561 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.79 2024-01-11T21:46:01.625 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.630 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.630 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.80 2024-01-11T21:46:01.690 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.693 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.693 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.81 2024-01-11T21:46:01.753 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:01.757 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:01.757 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.82 2024-01-11T21:46:02.030 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.035 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.035 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.83 2024-01-11T21:46:02.097 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.101 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.101 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.84 2024-01-11T21:46:02.159 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.163 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.163 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.85 2024-01-11T21:46:02.225 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.230 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.231 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.86 2024-01-11T21:46:02.293 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.298 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.298 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.87 2024-01-11T21:46:02.358 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.362 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.363 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.88 2024-01-11T21:46:02.426 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.431 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.431 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.89 2024-01-11T21:46:02.496 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.500 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.501 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.90 2024-01-11T21:46:02.568 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.573 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.573 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.91 2024-01-11T21:46:02.637 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.642 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.642 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.92 2024-01-11T21:46:02.704 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.709 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.709 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.93 2024-01-11T21:46:02.771 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:02.775 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:02.775 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.94 2024-01-11T21:46:03.046 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:03.050 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.050 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.95 2024-01-11T21:46:03.107 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:03.112 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.112 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.96 2024-01-11T21:46:03.176 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:03.181 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.181 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.97 2024-01-11T21:46:03.240 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:03.244 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.244 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.98 2024-01-11T21:46:03.309 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:03.313 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.313 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.99 2024-01-11T21:46:03.369 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:03.375 INFO:tasks.workunit.client.0.smithi178.stderr:++ seq -w 00 99 2024-01-11T21:46:03.376 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.376 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.00 --image-format 2 -s 1 2024-01-11T21:46:03.444 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.444 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.01 --image-format 2 -s 1 2024-01-11T21:46:03.504 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.504 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.02 --image-format 2 -s 1 2024-01-11T21:46:03.565 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.565 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.03 --image-format 2 -s 1 2024-01-11T21:46:03.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.04 --image-format 2 -s 1 2024-01-11T21:46:03.690 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.691 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.05 --image-format 2 -s 1 2024-01-11T21:46:03.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.06 --image-format 2 -s 1 2024-01-11T21:46:03.819 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.819 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.07 --image-format 2 -s 1 2024-01-11T21:46:03.880 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.881 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.08 --image-format 2 -s 1 2024-01-11T21:46:03.941 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.941 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.09 --image-format 2 -s 1 2024-01-11T21:46:03.998 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:03.998 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.10 --image-format 2 -s 1 2024-01-11T21:46:04.060 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.060 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.11 --image-format 2 -s 1 2024-01-11T21:46:04.121 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.121 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.12 --image-format 2 -s 1 2024-01-11T21:46:04.178 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.178 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.13 --image-format 2 -s 1 2024-01-11T21:46:04.237 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.237 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.14 --image-format 2 -s 1 2024-01-11T21:46:04.303 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.303 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.15 --image-format 2 -s 1 2024-01-11T21:46:04.369 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.369 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.16 --image-format 2 -s 1 2024-01-11T21:46:04.436 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.436 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.17 --image-format 2 -s 1 2024-01-11T21:46:04.501 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.501 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.18 --image-format 2 -s 1 2024-01-11T21:46:04.560 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.561 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.19 --image-format 2 -s 1 2024-01-11T21:46:04.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.20 --image-format 2 -s 1 2024-01-11T21:46:04.684 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.684 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.21 --image-format 2 -s 1 2024-01-11T21:46:04.748 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.748 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.22 --image-format 2 -s 1 2024-01-11T21:46:04.810 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.810 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.23 --image-format 2 -s 1 2024-01-11T21:46:04.876 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.876 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.24 --image-format 2 -s 1 2024-01-11T21:46:04.936 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.936 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.25 --image-format 2 -s 1 2024-01-11T21:46:04.989 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:04.989 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.26 --image-format 2 -s 1 2024-01-11T21:46:05.053 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.053 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.27 --image-format 2 -s 1 2024-01-11T21:46:05.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.28 --image-format 2 -s 1 2024-01-11T21:46:05.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.29 --image-format 2 -s 1 2024-01-11T21:46:05.233 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.233 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.30 --image-format 2 -s 1 2024-01-11T21:46:05.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.31 --image-format 2 -s 1 2024-01-11T21:46:05.350 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.350 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.32 --image-format 2 -s 1 2024-01-11T21:46:05.411 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.411 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.33 --image-format 2 -s 1 2024-01-11T21:46:05.474 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.474 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.34 --image-format 2 -s 1 2024-01-11T21:46:05.536 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.537 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.35 --image-format 2 -s 1 2024-01-11T21:46:05.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.36 --image-format 2 -s 1 2024-01-11T21:46:05.661 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.661 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.37 --image-format 2 -s 1 2024-01-11T21:46:05.719 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.719 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.38 --image-format 2 -s 1 2024-01-11T21:46:05.786 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.786 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.39 --image-format 2 -s 1 2024-01-11T21:46:05.847 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.847 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.40 --image-format 2 -s 1 2024-01-11T21:46:05.910 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.910 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.41 --image-format 2 -s 1 2024-01-11T21:46:05.967 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:05.967 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.42 --image-format 2 -s 1 2024-01-11T21:46:06.030 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.031 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.43 --image-format 2 -s 1 2024-01-11T21:46:06.090 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.090 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.44 --image-format 2 -s 1 2024-01-11T21:46:06.152 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.152 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.45 --image-format 2 -s 1 2024-01-11T21:46:06.213 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.213 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.46 --image-format 2 -s 1 2024-01-11T21:46:06.279 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.279 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.47 --image-format 2 -s 1 2024-01-11T21:46:06.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.48 --image-format 2 -s 1 2024-01-11T21:46:06.400 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.400 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.49 --image-format 2 -s 1 2024-01-11T21:46:06.460 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.460 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.50 --image-format 2 -s 1 2024-01-11T21:46:06.515 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.516 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.51 --image-format 2 -s 1 2024-01-11T21:46:06.576 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.577 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.52 --image-format 2 -s 1 2024-01-11T21:46:06.641 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.641 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.53 --image-format 2 -s 1 2024-01-11T21:46:06.706 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.706 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.54 --image-format 2 -s 1 2024-01-11T21:46:06.768 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.768 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.55 --image-format 2 -s 1 2024-01-11T21:46:06.824 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.824 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.56 --image-format 2 -s 1 2024-01-11T21:46:06.883 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.883 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.57 --image-format 2 -s 1 2024-01-11T21:46:06.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:06.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.58 --image-format 2 -s 1 2024-01-11T21:46:07.010 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.010 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.59 --image-format 2 -s 1 2024-01-11T21:46:07.075 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.075 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.60 --image-format 2 -s 1 2024-01-11T21:46:07.303 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:07.300+0000 7fae8e654700 0 --2- 172.21.15.178:0/3100774484 >> v2:172.21.15.178:3300/0 conn(0x55611f1313d0 0x55611f1317a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:46:07.347 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.347 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.61 --image-format 2 -s 1 2024-01-11T21:46:07.405 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.405 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.62 --image-format 2 -s 1 2024-01-11T21:46:07.467 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.467 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.63 --image-format 2 -s 1 2024-01-11T21:46:07.531 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.531 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.64 --image-format 2 -s 1 2024-01-11T21:46:07.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.65 --image-format 2 -s 1 2024-01-11T21:46:07.655 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.655 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.66 --image-format 2 -s 1 2024-01-11T21:46:07.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.67 --image-format 2 -s 1 2024-01-11T21:46:07.786 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.786 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.68 --image-format 2 -s 1 2024-01-11T21:46:07.849 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.849 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.69 --image-format 2 -s 1 2024-01-11T21:46:07.910 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.910 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.70 --image-format 2 -s 1 2024-01-11T21:46:07.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:07.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.71 --image-format 2 -s 1 2024-01-11T21:46:08.040 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.040 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.72 --image-format 2 -s 1 2024-01-11T21:46:08.108 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.108 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.73 --image-format 2 -s 1 2024-01-11T21:46:08.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.74 --image-format 2 -s 1 2024-01-11T21:46:08.236 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.236 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.75 --image-format 2 -s 1 2024-01-11T21:46:08.301 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.301 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.76 --image-format 2 -s 1 2024-01-11T21:46:08.358 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.359 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.77 --image-format 2 -s 1 2024-01-11T21:46:08.425 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.425 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.78 --image-format 2 -s 1 2024-01-11T21:46:08.689 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.689 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.79 --image-format 2 -s 1 2024-01-11T21:46:08.753 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:08.754 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.80 --image-format 2 -s 1 2024-01-11T21:46:09.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.81 --image-format 2 -s 1 2024-01-11T21:46:09.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.82 --image-format 2 -s 1 2024-01-11T21:46:09.161 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.161 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.83 --image-format 2 -s 1 2024-01-11T21:46:09.226 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.226 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.84 --image-format 2 -s 1 2024-01-11T21:46:09.286 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.286 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.85 --image-format 2 -s 1 2024-01-11T21:46:09.352 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.352 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.86 --image-format 2 -s 1 2024-01-11T21:46:09.410 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.410 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.87 --image-format 2 -s 1 2024-01-11T21:46:09.476 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.476 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.88 --image-format 2 -s 1 2024-01-11T21:46:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.89 --image-format 2 -s 1 2024-01-11T21:46:09.598 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.598 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.90 --image-format 2 -s 1 2024-01-11T21:46:09.660 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.660 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.91 --image-format 2 -s 1 2024-01-11T21:46:09.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.92 --image-format 2 -s 1 2024-01-11T21:46:09.799 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.800 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.93 --image-format 2 -s 1 2024-01-11T21:46:09.860 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.860 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.94 --image-format 2 -s 1 2024-01-11T21:46:09.926 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.926 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.95 --image-format 2 -s 1 2024-01-11T21:46:09.988 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:09.988 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.96 --image-format 2 -s 1 2024-01-11T21:46:10.057 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:10.057 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.97 --image-format 2 -s 1 2024-01-11T21:46:10.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:10.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.98 --image-format 2 -s 1 2024-01-11T21:46:10.182 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:10.182 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create image.99 --image-format 2 -s 1 2024-01-11T21:46:10.242 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:46:10.242 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:46:10.242 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 100 2024-01-11T21:46:10.282 INFO:tasks.workunit.client.0.smithi178.stdout:100 2024-01-11T21:46:10.283 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:46:10.284 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep image 2024-01-11T21:46:10.284 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:46:10.284 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 100 2024-01-11T21:46:10.356 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:10.352+0000 7f3fcb2a7700 0 -- 172.21.15.178:0/2520109827 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5570c48baf90 msgr2=0x5570c48bd370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:10.363 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:10.356+0000 7f3fcbaa8700 0 -- 172.21.15.178:0/2520109827 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5570c48a2180 msgr2=0x5570c48b2630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:10.717 INFO:tasks.workunit.client.0.smithi178.stdout:100 2024-01-11T21:46:10.717 INFO:tasks.workunit.client.0.smithi178.stderr:++ seq -w 00 99 2024-01-11T21:46:10.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:10.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.00 2024-01-11T21:46:10.869 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:10.864+0000 7f765923a700 0 -- 172.21.15.178:0/1072038176 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f763c036d10 msgr2=0x7f763c0390f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:10.883 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:10.888 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:10.889 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.01 2024-01-11T21:46:11.073 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:11.068+0000 7f2cef9c2700 0 -- 172.21.15.178:0/560969572 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f2cd4008740 msgr2=0x7f2cd400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:11.075 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:11.080 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:11.080 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.02 2024-01-11T21:46:11.253 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:11.248+0000 7ff7952d2700 0 -- 172.21.15.178:0/4084793730 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7ff770006520 msgr2=0x7ff770011ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:11.253 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:11.258 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:11.258 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.03 2024-01-11T21:46:11.427 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:11.420+0000 7fb1c4134700 0 -- 172.21.15.178:0/1063517550 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fb1a8008740 msgr2=0x7fb1a800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:11.428 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:11.432 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:11.432 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.04 2024-01-11T21:46:11.593 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:11.588+0000 7fa4701d1700 0 -- 172.21.15.178:0/1084599270 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55e2ec1b8be0 msgr2=0x55e2ec19e3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:11.602 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:11.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:11.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.05 2024-01-11T21:46:11.768 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:11.760+0000 7f5e25b52700 0 -- 172.21.15.178:0/431730283 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f5e0c008740 msgr2=0x7f5e0c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:11.768 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:11.772 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:11.772 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.06 2024-01-11T21:46:11.944 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:11.940+0000 7f965d286700 0 -- 172.21.15.178:0/1471491000 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x561ba305d0a0 msgr2=0x561ba30610f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:11.945 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:11.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:11.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.07 2024-01-11T21:46:12.126 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:12.120+0000 7f4b518c7700 0 -- 172.21.15.178:0/86877468 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x559af7fd3440 msgr2=0x559af7fcc910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:12.127 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:12.131 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:12.131 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.08 2024-01-11T21:46:12.313 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:12.308+0000 7fcdd43c8700 0 -- 172.21.15.178:0/2241669031 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fcdb8006520 msgr2=0x7fcdb8011ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:12.313 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:12.318 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:12.318 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.09 2024-01-11T21:46:12.490 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:12.484+0000 7f3d7efa1700 0 -- 172.21.15.178:0/518139807 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f3d64008740 msgr2=0x7f3d6400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:12.492 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:12.497 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:12.497 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.10 2024-01-11T21:46:12.668 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:12.664+0000 7fdaa4bc4700 0 -- 172.21.15.178:0/1871024750 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fda7c008740 msgr2=0x7fda7c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:12.670 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:12.674 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:12.674 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.11 2024-01-11T21:46:12.848 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:12.844+0000 7fd84e20a700 0 -- 172.21.15.178:0/1051273286 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fd8280060f0 msgr2=0x7fd8280085a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:12.849 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:12.855 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:12.855 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.12 2024-01-11T21:46:13.006 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:13.000+0000 7f765f938700 0 -- 172.21.15.178:0/2454388483 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f76380060e0 msgr2=0x7f7638008590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:13.007 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:13.013 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:13.013 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.13 2024-01-11T21:46:13.197 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:13.192+0000 7fcb09d15700 0 -- 172.21.15.178:0/326386671 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fcae8008740 msgr2=0x7fcae800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:13.198 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:13.203 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:13.203 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.14 2024-01-11T21:46:13.387 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:13.380+0000 7fd078cc1700 0 -- 172.21.15.178:0/2677999306 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fd054008740 msgr2=0x7fd05400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:13.389 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:13.395 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:13.395 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.15 2024-01-11T21:46:13.581 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:13.576+0000 7ff6913ec700 0 -- 172.21.15.178:0/2734978057 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55d9d45bfbe0 msgr2=0x55d9d459bb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:13.582 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:13.587 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:13.587 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.16 2024-01-11T21:46:13.765 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:13.760+0000 7f1b59c9d700 0 -- 172.21.15.178:0/4291066525 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55c81ac02dd0 msgr2=0x55c81ac00710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:13.766 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:13.770 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:13.770 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.17 2024-01-11T21:46:13.940 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:13.936+0000 7f1a6546b700 0 -- 172.21.15.178:0/2448275 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f1a40006520 msgr2=0x7f1a40011ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:13.940 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:13.944 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:13.944 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.18 2024-01-11T21:46:14.121 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:14.116+0000 7ff4543da700 0 -- 172.21.15.178:0/1809232079 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5555bee0d9d0 msgr2=0x5555bee0cf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:14.121 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:14.125 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:14.125 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.19 2024-01-11T21:46:14.302 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:14.296+0000 7f9f63158700 0 -- 172.21.15.178:0/2611728651 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f9f480088b0 msgr2=0x7f9f4800ad30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:14.304 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:14.310 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:14.310 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.20 2024-01-11T21:46:14.489 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:14.484+0000 7f13e71ac700 0 -- 172.21.15.178:0/1307195883 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55570b317940 msgr2=0x55570b2cc990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:14.527 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:14.528 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:14.528 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.21 2024-01-11T21:46:14.665 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:14.660+0000 7ff23d572700 0 -- 172.21.15.178:0/2084295160 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x556fd8a0ff60 msgr2=0x556fd89e78d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:14.665 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:14.670 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:14.670 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.22 2024-01-11T21:46:14.837 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:14.832+0000 7f8dc538c700 0 -- 172.21.15.178:0/3520942612 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f8da0008740 msgr2=0x7f8da000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:14.839 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:14.843 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:14.843 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.23 2024-01-11T21:46:15.018 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:15.012+0000 7f7f98855700 0 -- 172.21.15.178:0/1959658043 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f7f780060e0 msgr2=0x7f7f78008590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:15.019 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:15.024 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:15.024 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.24 2024-01-11T21:46:15.193 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:15.188+0000 7f8f7b981700 0 -- 172.21.15.178:0/1757924187 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f8f58008740 msgr2=0x7f8f5800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:15.194 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:15.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:15.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.25 2024-01-11T21:46:15.348 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:15.344+0000 7fa36cf89700 0 -- 172.21.15.178:0/133797179 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x559bec57af20 msgr2=0x7fa350039a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:15.374 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:15.380 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:15.380 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.26 2024-01-11T21:46:15.558 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:15.552+0000 7fdb3cbca700 0 -- 172.21.15.178:0/2691893660 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fdb20008740 msgr2=0x7fdb2000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:15.559 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:15.562 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:15.563 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.27 2024-01-11T21:46:15.720 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:15.716+0000 7f64cf2b2700 0 -- 172.21.15.178:0/2740204090 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f64b4008740 msgr2=0x7f64b400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:15.722 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:15.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:15.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.28 2024-01-11T21:46:15.895 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:15.888+0000 7f3db5708700 0 -- 172.21.15.178:0/1339529426 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f3d94006740 msgr2=0x7f3d94008bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:15.896 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:15.899 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:15.899 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.29 2024-01-11T21:46:16.057 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:16.052+0000 7fa951088700 0 -- 172.21.15.178:0/598238744 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fa92c008740 msgr2=0x7fa92c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:16.058 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:16.063 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:16.063 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.30 2024-01-11T21:46:16.235 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:16.228+0000 7fd4f8d4a700 0 -- 172.21.15.178:0/1282574032 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x559f16ba9020 msgr2=0x559f16c4b9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:16.243 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:16.247 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:16.247 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.31 2024-01-11T21:46:16.421 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:16.416+0000 7fc60567b700 0 -- 172.21.15.178:0/1664761328 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fc5e0008740 msgr2=0x7fc5e000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:16.423 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:16.428 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:16.428 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.32 2024-01-11T21:46:16.606 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:16.600+0000 7f066c196700 0 -- 172.21.15.178:0/3511796343 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x560a97250d60 msgr2=0x560a9722c9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:16.607 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:16.611 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:16.611 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.33 2024-01-11T21:46:16.787 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:16.780+0000 7f3c6f158700 0 -- 172.21.15.178:0/3106331863 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55a269dbee60 msgr2=0x55a269d959a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:16.787 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:16.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:16.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.34 2024-01-11T21:46:16.965 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:16.960+0000 7f87276b2700 0 -- 172.21.15.178:0/3883052937 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f87000064e0 msgr2=0x7f87000088c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:16.965 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:16.969 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:16.969 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.35 2024-01-11T21:46:17.145 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:17.140+0000 7f52a4702700 0 -- 172.21.15.178:0/1118209481 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f5288008740 msgr2=0x7f528800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:17.147 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:17.152 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:17.152 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.36 2024-01-11T21:46:17.333 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:17.328+0000 7f4f4cc8d700 0 -- 172.21.15.178:0/1003031231 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5571db3651c0 msgr2=0x5571db31aa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:17.333 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:17.337 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:17.337 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.37 2024-01-11T21:46:17.522 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:17.516+0000 7f5770750700 0 -- 172.21.15.178:0/4012334285 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55bd9788a210 msgr2=0x55bd97861660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:17.522 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:17.527 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:17.527 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.38 2024-01-11T21:46:17.698 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:17.692+0000 7f88488fd700 0 -- 172.21.15.178:0/513378906 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f882c036b70 msgr2=0x7f882c038f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:17.714 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:17.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:17.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.39 2024-01-11T21:46:17.893 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:17.888+0000 7faf09d31700 0 -- 172.21.15.178:0/1434958033 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7faee8006560 msgr2=0x7faee8011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:17.894 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:17.899 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:17.899 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.40 2024-01-11T21:46:18.073 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:18.068+0000 7fdf30b45700 0 -- 172.21.15.178:0/3409516681 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fdf10036c40 msgr2=0x7fdf10039020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:18.087 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:18.091 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:18.091 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.41 2024-01-11T21:46:18.281 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:18.276+0000 7f4334fe3700 0 -- 172.21.15.178:0/1035772234 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f4310006520 msgr2=0x7f4310011ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:18.281 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:18.285 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:18.285 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.42 2024-01-11T21:46:18.456 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:18.452+0000 7fb3f189a700 0 -- 172.21.15.178:0/674451064 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fb3d0008740 msgr2=0x7fb3d000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:18.458 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:18.462 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:18.462 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.43 2024-01-11T21:46:18.632 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:18.624+0000 7f740357a700 0 -- 172.21.15.178:0/4129863706 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55aa56063550 msgr2=0x55aa5603b630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:18.632 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:18.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:18.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.44 2024-01-11T21:46:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:18.812+0000 7fea26aab700 0 -- 172.21.15.178:0/4001980654 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fea04008740 msgr2=0x7fea0400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:18.820 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:18.825 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:18.825 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.45 2024-01-11T21:46:19.005 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:19.000+0000 7f8e544f5700 0 -- 172.21.15.178:0/3653462425 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f8e38008740 msgr2=0x7f8e3800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:19.005 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:19.009 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:19.010 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.46 2024-01-11T21:46:19.196 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:19.188+0000 7fc4c1884700 0 -- 172.21.15.178:0/2801397645 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5590cfd4e880 msgr2=0x5590cfd4f5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:19.196 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:19.200 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:19.200 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.47 2024-01-11T21:46:19.374 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:19.368+0000 7f09fa93d700 0 -- 172.21.15.178:0/722755844 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x559c08b07f90 msgr2=0x559c08ae0570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:19.375 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:19.380 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:19.380 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.48 2024-01-11T21:46:19.555 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:19.548+0000 7f854f1fa700 0 -- 172.21.15.178:0/2941147245 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f8528005fc0 msgr2=0x7f8528008470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:19.555 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:19.559 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:19.559 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.49 2024-01-11T21:46:19.725 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:19.720+0000 7f67229c6700 0 -- 172.21.15.178:0/2010977568 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f6708036b60 msgr2=0x7f6708038f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:19.748 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:19.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:19.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.50 2024-01-11T21:46:19.932 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:19.924+0000 7fd1b095e700 0 -- 172.21.15.178:0/1598689113 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fd18c008740 msgr2=0x7fd18c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:19.932 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:19.937 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:19.937 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.51 2024-01-11T21:46:20.124 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:20.120+0000 7f2cfc011700 0 -- 172.21.15.178:0/899486836 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f2ce0008740 msgr2=0x7f2ce000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:20.128 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:20.133 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:20.133 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.52 2024-01-11T21:46:20.315 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:20.308+0000 7f9482016700 0 -- 172.21.15.178:0/1780999461 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f945c0060e0 msgr2=0x7f945c008590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:20.316 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:20.321 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:20.321 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.53 2024-01-11T21:46:20.496 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:20.492+0000 7f6e2893f700 0 -- 172.21.15.178:0/842621333 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f6e0c008740 msgr2=0x7f6e0c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:20.498 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:20.503 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:20.503 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.54 2024-01-11T21:46:20.667 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:20.660+0000 7efd38750700 0 -- 172.21.15.178:0/282554015 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55b0b1aab020 msgr2=0x55b0b1b4da20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:20.676 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:20.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:20.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.55 2024-01-11T21:46:20.850 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:20.844+0000 7f526517f700 0 -- 172.21.15.178:0/1704558236 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f5240008740 msgr2=0x7f524000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:20.850 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:20.855 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:20.855 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.56 2024-01-11T21:46:21.039 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:21.032+0000 7f868e8cb700 0 -- 172.21.15.178:0/1788667048 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f866c008740 msgr2=0x7f866c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:21.040 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:21.044 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:21.044 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.57 2024-01-11T21:46:21.231 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:21.224+0000 7ff9c1136700 0 -- 172.21.15.178:0/1078458924 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7ff99c006560 msgr2=0x7ff99c011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:21.231 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:21.236 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:21.236 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.58 2024-01-11T21:46:21.399 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:21.392+0000 7fb5157bb700 0 -- 172.21.15.178:0/4037547312 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fb4f8036b70 msgr2=0x7fb4f8038f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:21.417 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:21.422 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:21.422 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.59 2024-01-11T21:46:21.601 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:21.596+0000 7f7046e54700 0 -- 172.21.15.178:0/4236755229 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f702c006560 msgr2=0x7f702c011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:21.602 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:21.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:21.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.60 2024-01-11T21:46:21.790 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:21.784+0000 7fbf0f4c7700 0 -- 172.21.15.178:0/4060196890 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55b40c5548b0 msgr2=0x55b40c576000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:21.792 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:21.797 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:21.797 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.61 2024-01-11T21:46:21.980 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:21.972+0000 7f91ca28b700 0 -- 172.21.15.178:0/2085301292 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f91ac036b70 msgr2=0x7f91ac038f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:22.187 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:22.193 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:22.193 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.62 2024-01-11T21:46:22.372 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:22.368+0000 7f58d6804700 0 -- 172.21.15.178:0/2189521952 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f58bc008740 msgr2=0x7f58bc00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:22.384 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:22.389 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:22.389 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.63 2024-01-11T21:46:22.573 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:22.568+0000 7f2004b55700 0 -- 172.21.15.178:0/3792902706 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f1fe4006110 msgr2=0x7f1fe40085c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:22.576 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:22.581 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:22.581 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.64 2024-01-11T21:46:22.754 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:22.748+0000 7fad5328c700 0 -- 172.21.15.178:0/2721140018 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fad38008740 msgr2=0x7fad3800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:22.759 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:22.764 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:22.764 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.65 2024-01-11T21:46:22.945 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:22.940+0000 7f7577565700 0 -- 172.21.15.178:0/2098414823 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x555966866850 msgr2=0x555966888c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:22.947 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:22.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:22.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.66 2024-01-11T21:46:23.119 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:23.112+0000 7f0b99119700 0 -- 172.21.15.178:0/521374171 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f0b74008740 msgr2=0x7f0b7400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:23.119 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:23.123 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:23.123 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.67 2024-01-11T21:46:23.308 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:23.304+0000 7f1af50e2700 0 -- 172.21.15.178:0/614631532 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f1ad0008740 msgr2=0x7f1ad000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:23.310 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:23.315 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:23.315 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.68 2024-01-11T21:46:23.501 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:23.496+0000 7f5667220700 0 -- 172.21.15.178:0/1295232316 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f5644008740 msgr2=0x7f564400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:23.501 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:23.505 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:23.505 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.69 2024-01-11T21:46:23.680 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:23.676+0000 7f0dafca1700 0 -- 172.21.15.178:0/3369570685 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f0d94008740 msgr2=0x7f0d9400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:23.681 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:23.686 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:23.687 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.70 2024-01-11T21:46:23.872 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:23.868+0000 7ff93a5b4700 0 -- 172.21.15.178:0/3810297923 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7ff920008740 msgr2=0x7ff92000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:23.874 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:23.879 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:23.879 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.71 2024-01-11T21:46:24.053 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:24.048+0000 7fb35c922700 0 -- 172.21.15.178:0/3041199581 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fb33c036b70 msgr2=0x7fb33c038f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:24.068 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:24.073 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:24.073 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.72 2024-01-11T21:46:24.455 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:24.448+0000 7ff85ffc1700 0 -- 172.21.15.178:0/2907795878 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7ff844008740 msgr2=0x7ff84400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:24.465 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:24.470 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:24.470 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.73 2024-01-11T21:46:24.650 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:24.644+0000 7ff69e7f4700 0 -- 172.21.15.178:0/1970952048 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5625bcc2bb70 msgr2=0x5625bcc2df50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:24.657 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:24.661 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:24.661 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.74 2024-01-11T21:46:24.849 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:24.844+0000 7f7d29f82700 0 -- 172.21.15.178:0/199093779 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f7d10008740 msgr2=0x7f7d1000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:24.849 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:24.854 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:24.854 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.75 2024-01-11T21:46:25.034 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:25.028+0000 7f140a86c700 0 -- 172.21.15.178:0/2894098675 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f13f0008740 msgr2=0x7f13f000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:25.034 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:25.039 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:25.039 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.76 2024-01-11T21:46:25.221 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:25.216+0000 7f731eafb700 0 -- 172.21.15.178:0/989718580 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f7304008740 msgr2=0x7f730400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:25.223 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:25.227 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:25.228 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.77 2024-01-11T21:46:25.410 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:25.404+0000 7fdbd48b6700 0 -- 172.21.15.178:0/3396646928 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55a8275e1790 msgr2=0x55a8275b9ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:25.411 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:25.417 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:25.417 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.78 2024-01-11T21:46:25.602 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:25.596+0000 7fde24d7c700 0 -- 172.21.15.178:0/2320248281 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fddfc006500 msgr2=0x7fddfc0128a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:25.604 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:25.608 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:25.608 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.79 2024-01-11T21:46:25.776 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:25.772+0000 7fabd739a700 0 -- 172.21.15.178:0/1671501547 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fabb8005f70 msgr2=0x7fabb8008420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:25.789 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:25.794 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:25.794 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.80 2024-01-11T21:46:25.993 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:25.988+0000 7f04ba49c700 0 -- 172.21.15.178:0/3845947723 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f04a0008740 msgr2=0x7f04a000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:25.993 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.81 2024-01-11T21:46:26.182 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:26.176+0000 7f98e3da2700 0 -- 172.21.15.178:0/524857960 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f98c0008740 msgr2=0x7f98c000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:26.182 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:26.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:26.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.82 2024-01-11T21:46:26.374 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:26.368+0000 7f8abb6f7700 0 -- 172.21.15.178:0/1252057796 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f8a94008740 msgr2=0x7f8a9400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:26.374 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:26.378 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:26.378 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.83 2024-01-11T21:46:26.559 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:26.552+0000 7f9fa9873700 0 -- 172.21.15.178:0/2023218174 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f9f840060c0 msgr2=0x7f9f84008570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:26.559 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:26.561 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:26.561 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.84 2024-01-11T21:46:26.745 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:26.740+0000 7f2c3bfd3700 0 -- 172.21.15.178:0/2574545797 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x558782f214b0 msgr2=0x558782ef86d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:26.746 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:26.751 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:26.751 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.85 2024-01-11T21:46:26.943 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:26.936+0000 7f1508f5c700 0 -- 172.21.15.178:0/1693991907 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f14ec008740 msgr2=0x7f14ec00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:26.943 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:26.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:26.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.86 2024-01-11T21:46:27.134 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:27.128+0000 7f023d870700 0 -- 172.21.15.178:0/2145325732 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x556349150980 msgr2=0x55634912f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:27.136 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:27.140 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:27.140 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.87 2024-01-11T21:46:27.326 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:27.320+0000 7f96bc7a6700 0 -- 172.21.15.178:0/3935573563 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f9694006090 msgr2=0x7f9694008540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:27.328 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:27.333 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:27.333 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.88 2024-01-11T21:46:27.510 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:27.504+0000 7f29deea6700 0 -- 172.21.15.178:0/2973050269 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f29c0036b70 msgr2=0x7f29c0038f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:27.522 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:27.526 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:27.526 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.89 2024-01-11T21:46:27.718 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:27.712+0000 7fcbbf8fe700 0 -- 172.21.15.178:0/799535563 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fcba4008740 msgr2=0x7fcba400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:27.719 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:27.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:27.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.90 2024-01-11T21:46:27.916 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:27.908+0000 7f4ceff7c700 0 -- 172.21.15.178:0/2659363262 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f4cd4008740 msgr2=0x7f4cd400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:27.916 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:27.920 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:27.920 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.91 2024-01-11T21:46:28.098 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:28.092+0000 7fe80fe00700 0 -- 172.21.15.178:0/1773641178 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fe7f4008740 msgr2=0x7fe7f400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:28.099 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:28.104 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:28.104 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.92 2024-01-11T21:46:28.287 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:28.280+0000 7fb7bbb7c700 0 -- 172.21.15.178:0/3565549251 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fb7a0006580 msgr2=0x7fb7a0011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:28.289 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:28.294 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:28.294 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.93 2024-01-11T21:46:28.484 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:28.480+0000 7fe243b6d700 0 -- 172.21.15.178:0/523829727 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fe220008740 msgr2=0x7fe22000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:28.484 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:28.489 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:28.489 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.94 2024-01-11T21:46:28.674 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:28.668+0000 7f1a8bfc1700 0 -- 172.21.15.178:0/3678326149 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f1a70008740 msgr2=0x7f1a7000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:28.675 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:28.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:28.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.95 2024-01-11T21:46:28.858 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:28.852+0000 7f62eff3c700 0 -- 172.21.15.178:0/2964520310 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55b14f113a50 msgr2=0x55b14f0f14d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:28.858 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:28.862 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:28.862 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.96 2024-01-11T21:46:29.052 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:29.048+0000 7ff573705700 0 -- 172.21.15.178:0/2397236490 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7ff558006580 msgr2=0x7ff558011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:29.052 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:29.057 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:29.057 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.97 2024-01-11T21:46:29.243 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:29.236+0000 7f355f32f700 0 -- 172.21.15.178:0/964972607 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x563eb8e964f0 msgr2=0x563eb8e67580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:29.244 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:29.248 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:29.249 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.98 2024-01-11T21:46:29.444 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:29.440+0000 7f3c1cf20700 0 -- 172.21.15.178:0/2189306059 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f3c00008740 msgr2=0x7f3c0000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:29.446 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:29.451 INFO:tasks.workunit.client.0.smithi178.stderr:+ for i in $(seq -w 00 99) 2024-01-11T21:46:29.451 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm image.99 2024-01-11T21:46:29.645 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:29.640+0000 7f103c2dd700 0 -- 172.21.15.178:0/1310018321 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55badf572b90 msgr2=0x55badf54a7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:29.647 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:29.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_remove 2024-01-11T21:46:29.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing remove...' 2024-01-11T21:46:29.653 INFO:tasks.workunit.client.0.smithi178.stdout:testing remove... 2024-01-11T21:46:29.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:46:29.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:29.772 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:29.901 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.021 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.140 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.391 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.516 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.648 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.777 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:30.903 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.025 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.154 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.275 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.411 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.522 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.643 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:31.891 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd remove NOT_EXIST 2024-01-11T21:46:31.959 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 0% complete...failed. 2024-01-11T21:46:31.959 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: delete error: (2) No such file or directory 2024-01-11T21:46:31.964 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:46:31.964 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 1 -s 1 test1 2024-01-11T21:46:31.992 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:46:32.014 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:32.008+0000 7fd1807ff4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:46:32.028 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:46:32.086 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:32.091 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:46:32.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:46:32.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:46:32.133 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:46:32.133 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test2 2024-01-11T21:46:32.202 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:46:32.389 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:32.384+0000 7efc549d9700 0 -- 172.21.15.178:0/1669455951 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55a820534ca0 msgr2=0x55a820511f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:32.390 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:46:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:46:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:46:32.437 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:46:32.437 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 1 -s 1 test1 2024-01-11T21:46:32.466 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:46:32.479 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:32.472+0000 7f8ee52634c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:46:32.494 INFO:tasks.workunit.client.0.smithi178.stderr:+ rados rm -p rbd test1.rbd 2024-01-11T21:46:32.539 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:46:32.588 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:32.593 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:46:32.593 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:46:32.593 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:46:32.637 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:46:32.638 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 1 -eq 0 ']' 2024-01-11T21:46:32.638 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test2 2024-01-11T21:46:32.710 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test2@snap 2024-01-11T21:46:33.544 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:46:33.553 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect test2@snap 2024-01-11T21:46:33.623 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone test2@snap clone --rbd-default-clone-format 1 2024-01-11T21:46:33.733 INFO:tasks.workunit.client.0.smithi178.stderr:+ rados -p rbd rm rbd_children 2024-01-11T21:46:33.774 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm clone 2024-01-11T21:46:33.962 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:33.956+0000 7fd972b52700 0 -- 172.21.15.178:0/3899068479 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fd958008740 msgr2=0x7fd95800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:33.966 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:33.972 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:46:33.973 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:46:33.973 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep clone 2024-01-11T21:46:33.973 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:46:34.015 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:46:34.015 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect test2@snap 2024-01-11T21:46:34.084 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm test2@snap 2024-01-11T21:46:34.534 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:46:34.548 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:46:34.708 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:34.704+0000 7f8ed0dc3700 0 -- 172.21.15.178:0/3065218554 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f8eb0036cd0 msgr2=0x7f8eb00390f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:34.729 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:34.734 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_migration 2024-01-11T21:46:34.734 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing migration...' 2024-01-11T21:46:34.734 INFO:tasks.workunit.client.0.smithi178.stdout:testing migration... 2024-01-11T21:46:34.735 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:46:34.735 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:34.859 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:34.980 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.107 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.227 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.355 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.478 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.707 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.834 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:35.959 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.083 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.212 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.338 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.469 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.591 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.844 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:46:36.972 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool create rbd2 8 2024-01-11T21:46:37.654 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd2' already exists 2024-01-11T21:46:37.662 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd pool init rbd2 2024-01-11T21:46:40.577 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 1 -s 128M test1 2024-01-11T21:46:40.601 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:46:40.613 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:40.612+0000 7fb7128054c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:46:40.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'format: 1' 2024-01-11T21:46:40.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info test1 2024-01-11T21:46:40.680 INFO:tasks.workunit.client.0.smithi178.stdout: format: 1 2024-01-11T21:46:40.681 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test1 --image-format 2 2024-01-11T21:46:40.808 INFO:tasks.workunit.client.0.smithi178.stderr:++ get_migration_state test1 2024-01-11T21:46:40.808 INFO:tasks.workunit.client.0.smithi178.stderr:++ local image=test1 2024-01-11T21:46:40.808 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd --format xml status test1 2024-01-11T21:46:40.809 INFO:tasks.workunit.client.0.smithi178.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-01-11T21:46:40.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ test prepared = prepared 2024-01-11T21:46:40.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info test1 2024-01-11T21:46:40.951 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'format: 2' 2024-01-11T21:46:41.014 INFO:tasks.workunit.client.0.smithi178.stdout: format: 2 2024-01-11T21:46:41.015 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:46:41.073 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:41.072+0000 7f86b86374c0 -1 librbd::image::PreRemoveRequest: 0x55963a45e4b0 validate_image_removal: image in migration state - not removing 2024-01-11T21:46:41.077 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 0% complete...failed. 2024-01-11T21:46:41.077 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: error: image still has watchers 2024-01-11T21:46:41.077 INFO:tasks.workunit.client.0.smithi178.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-01-11T21:46:41.081 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:46:41.081 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test1 2024-01-11T21:46:41.263 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete...2024-01-11T21:46:41.260+0000 7fca4544a700 0 -- 172.21.15.178:0/3370915352 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5624d4102150 msgr2=0x5624d41f5580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:41.287 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:46:41.284+0000 7fca4544a700 0 -- 172.21.15.178:0/3370915352 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fca24037370 msgr2=0x7fca24039790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:41.315 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:41.318 INFO:tasks.workunit.client.0.smithi178.stderr:++ get_migration_state test1 2024-01-11T21:46:41.319 INFO:tasks.workunit.client.0.smithi178.stderr:++ local image=test1 2024-01-11T21:46:41.319 INFO:tasks.workunit.client.0.smithi178.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-01-11T21:46:41.319 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd --format xml status test1 2024-01-11T21:46:41.421 INFO:tasks.workunit.client.0.smithi178.stderr:+ test executed = executed 2024-01-11T21:46:41.422 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 2024-01-11T21:46:41.595 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete...2024-01-11T21:46:41.592+0000 7fe09dd94700 0 -- 172.21.15.178:0/3849790211 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55f032898790 msgr2=0x55f032985cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:41.608 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2024-01-11T21:46:41.608+0000 7fe09dd94700 0 -- 172.21.15.178:0/3849790211 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fe080037440 msgr2=0x7fe080039860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:41.620 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 100% complete...done. 2024-01-11T21:46:41.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ get_migration_state test1 2024-01-11T21:46:41.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ local image=test1 2024-01-11T21:46:41.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --format xml status test1 2024-01-11T21:46:41.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ xmlstarlet sel -t -v //status/migration/state 2024-01-11T21:46:41.678 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:46:41.679 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'features: .*layering' 2024-01-11T21:46:41.679 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info test1 2024-01-11T21:46:41.740 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:46:41.740 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test1 --image-feature layering,exclusive-lock,object-map,fast-diff,deep-flatten 2024-01-11T21:46:41.926 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info test1 2024-01-11T21:46:41.927 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'features: .*layering' 2024-01-11T21:46:42.000 INFO:tasks.workunit.client.0.smithi178.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, migrating 2024-01-11T21:46:42.001 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test1 2024-01-11T21:46:42.162 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:46:42.160+0000 7ff9bac43700 0 -- 172.21.15.178:0/3929891793 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7ff9a0008740 msgr2=0x7ff9a000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:42.194 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:42.192+0000 7ff9bac43700 0 -- 172.21.15.178:0/3929891793 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x558de7503150 msgr2=0x7ff99c03c4e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:42.200 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:42.203 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 2024-01-11T21:46:42.366 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:42.364+0000 7efede37f700 0 -- 172.21.15.178:0/244078177 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7efec0037510 msgr2=0x7efec0039930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:42.389 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete...2024-01-11T21:46:42.388+0000 7efede37f700 0 -- 172.21.15.178:0/244078177 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7efec4008740 msgr2=0x7efec003ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:42.423 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2024-01-11T21:46:42.429 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test1 rbd2/test1 2024-01-11T21:46:42.606 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:42.604+0000 7efeb04a6700 0 -- 172.21.15.178:0/754042026 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7efe94008740 msgr2=0x7efe9400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:42.619 INFO:tasks.workunit.client.0.smithi178.stderr:++ get_migration_state rbd2/test1 2024-01-11T21:46:42.620 INFO:tasks.workunit.client.0.smithi178.stderr:++ local image=rbd2/test1 2024-01-11T21:46:42.620 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd --format xml status rbd2/test1 2024-01-11T21:46:42.620 INFO:tasks.workunit.client.0.smithi178.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-01-11T21:46:42.770 INFO:tasks.workunit.client.0.smithi178.stderr:+ test prepared = prepared 2024-01-11T21:46:42.771 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:46:42.771 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:46:42.771 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:46:42.814 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:46:42.815 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd -p rbd2 ls 2024-01-11T21:46:42.815 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:46:42.857 INFO:tasks.workunit.client.0.smithi178.stdout:test1 2024-01-11T21:46:42.858 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test1 2024-01-11T21:46:43.017 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:46:43.016+0000 7faaadb69700 0 -- 172.21.15.178:0/3370701869 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55b6b74f3620 msgr2=0x55b6b74f5a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:43.027 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:43.031 INFO:tasks.workunit.client.0.smithi178.stderr:++ get_migration_state rbd2/test1 2024-01-11T21:46:43.031 INFO:tasks.workunit.client.0.smithi178.stderr:++ local image=rbd2/test1 2024-01-11T21:46:43.032 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd --format xml status rbd2/test1 2024-01-11T21:46:43.032 INFO:tasks.workunit.client.0.smithi178.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-01-11T21:46:43.144 INFO:tasks.workunit.client.0.smithi178.stderr:+ test executed = executed 2024-01-11T21:46:43.145 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm rbd2/test1 2024-01-11T21:46:43.209 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:43.208+0000 7f4132e82700 -1 librbd::image::PreRemoveRequest: 0x55e05dc65160 validate_image_removal: image in migration state - not removing 2024-01-11T21:46:43.218 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 0% complete...failed. 2024-01-11T21:46:43.218 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: error: image still has watchers 2024-01-11T21:46:43.218 INFO:tasks.workunit.client.0.smithi178.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-01-11T21:46:43.224 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:46:43.224 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 2024-01-11T21:46:43.385 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2024-01-11T21:46:43.384+0000 7f1a6e512700 0 -- 172.21.15.178:0/3394140133 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f1a50037370 msgr2=0x7f1a50039790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:43.420 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 100% complete...done. 2024-01-11T21:46:43.425 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace create rbd2/ns1 2024-01-11T21:46:43.471 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace create rbd2/ns2 2024-01-11T21:46:43.518 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare rbd2/test1 rbd2/ns1/test1 2024-01-11T21:46:43.678 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:43.676+0000 7fe15a6b6700 0 -- 172.21.15.178:0/2296460351 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fe140008740 msgr2=0x7fe14000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:43.690 INFO:tasks.workunit.client.0.smithi178.stderr:++ get_migration_state rbd2/ns1/test1 2024-01-11T21:46:43.690 INFO:tasks.workunit.client.0.smithi178.stderr:++ local image=rbd2/ns1/test1 2024-01-11T21:46:43.690 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd --format xml status rbd2/ns1/test1 2024-01-11T21:46:43.690 INFO:tasks.workunit.client.0.smithi178.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-01-11T21:46:43.841 INFO:tasks.workunit.client.0.smithi178.stderr:+ test prepared = prepared 2024-01-11T21:46:43.842 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute rbd2/test1 2024-01-11T21:46:43.975 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:46:43.972+0000 7fe99d4c3700 0 -- 172.21.15.178:0/1732831537 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x561b5ce45cc0 msgr2=0x561b5ce480a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:43.980 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:43.985 INFO:tasks.workunit.client.0.smithi178.stderr:++ get_migration_state rbd2/ns1/test1 2024-01-11T21:46:43.985 INFO:tasks.workunit.client.0.smithi178.stderr:++ local image=rbd2/ns1/test1 2024-01-11T21:46:43.985 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd --format xml status rbd2/ns1/test1 2024-01-11T21:46:43.985 INFO:tasks.workunit.client.0.smithi178.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-01-11T21:46:44.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ test executed = executed 2024-01-11T21:46:44.092 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit rbd2/test1 2024-01-11T21:46:44.242 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2024-01-11T21:46:44.240+0000 7fa4bd8b4700 0 -- 172.21.15.178:0/1104330868 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x56288a3b5260 msgr2=0x56288a3b7640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:44.258 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 100% complete...done. 2024-01-11T21:46:44.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare rbd2/ns1/test1 rbd2/ns2/test1 2024-01-11T21:46:44.421 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:44.420+0000 7f84d847c700 0 -- 172.21.15.178:0/2742315565 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x56049f98b330 msgr2=0x56049f98d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:44.432 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute rbd2/ns2/test1 2024-01-11T21:46:44.549 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:46:44.548+0000 7f0e9ee1a700 0 -- 172.21.15.178:0/1557294054 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55f9b69b1ae0 msgr2=0x55f9b6a5b0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:44.562 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:44.560+0000 7f0e9ee1a700 0 -- 172.21.15.178:0/1557294054 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f0e80037370 msgr2=0x7f0e80039750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:44.565 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:44.569 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit rbd2/ns2/test1 2024-01-11T21:46:44.720 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:44.720+0000 7f1576675700 0 -- 172.21.15.178:0/3183666138 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x557353ae03d0 msgr2=0x557353b84fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:44.736 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:44.736+0000 7f1576675700 0 -- 172.21.15.178:0/3183666138 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f15580372f0 msgr2=0x7f1558039710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:44.761 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2024-01-11T21:46:44.766 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M test1 2024-01-11T21:46:44.804 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:44.804+0000 7f6c61d4e4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:46:44.818 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test1 --data-pool rbd2 2024-01-11T21:46:44.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info test1 2024-01-11T21:46:44.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'data_pool: rbd2' 2024-01-11T21:46:45.010 INFO:tasks.workunit.client.0.smithi178.stdout: data_pool: rbd2 2024-01-11T21:46:45.010 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test1 2024-01-11T21:46:45.160 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete...2024-01-11T21:46:45.160+0000 7fcc9e833700 0 -- 172.21.15.178:0/697830961 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x559f554931f0 msgr2=0x559f55584350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:45.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:46:45.168+0000 7f6c30961700 -1 reset not still connected to 0x564c34c3e750 2024-01-11T21:46:45.181 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:46:45.180+0000 7fcc9e833700 0 -- 172.21.15.178:0/697830961 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fcc78004be0 msgr2=0x7fcc8003c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:45.201 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:45.290 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 2024-01-11T21:46:45.369 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete...2024-01-11T21:46:45.368+0000 7f2b35b25700 0 -- 172.21.15.178:0/2986374939 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f2b10004be0 msgr2=0x7f2b10006fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:45.380 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2024-01-11T21:46:45.380+0000 7f2b36326700 0 -- 172.21.15.178:0/2986374939 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55e073fdd4e0 msgr2=0x55e0740ca9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:45.394 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 100% complete...done. 2024-01-11T21:46:45.398 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test1 2024-01-11T21:46:45.553 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd trash mv test1 2024-01-11T21:46:45.554 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv test1 2024-01-11T21:46:45.612 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:45.608+0000 7f554a6e24c0 -1 librbd::api::Trash: move: cannot move migrating image to trash 2024-01-11T21:46:45.615 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: deferred delete error: (16) Device or resource busy 2024-01-11T21:46:45.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:46:45.621 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls -a 2024-01-11T21:46:45.621 INFO:tasks.workunit.client.0.smithi178.stderr:++ cut -d ' ' -f 1 2024-01-11T21:46:45.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=19e7f00118fa 2024-01-11T21:46:45.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd trash rm 19e7f00118fa 2024-01-11T21:46:45.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash rm 19e7f00118fa 2024-01-11T21:46:45.716 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:45.712+0000 7fd6b0d94700 -1 librbd::image::RefreshRequest: image being migrated 2024-01-11T21:46:45.716 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:45.712+0000 7fd6b0d94700 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2024-01-11T21:46:45.716 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:45.712+0000 7fd6abfff700 -1 librbd::ImageState: 0x7fd68c04d9f0 failed to open image: (30) Read-only file system 2024-01-11T21:46:45.716 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:45.716+0000 7fd692ffd700 -1 librbd::image::RemoveRequest: 0x7fd68c000b80 handle_open_image: error opening image: (30) Read-only file system 2024-01-11T21:46:45.720 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: remove error: (30) Read-only file system 2024-01-11T21:46:45.720 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 0% complete...failed. 2024-01-11T21:46:45.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:46:45.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd trash restore 19e7f00118fa 2024-01-11T21:46:45.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash restore 19e7f00118fa 2024-01-11T21:46:45.763 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: restore error: 2024-01-11T21:46:45.760+0000 7f4cc8e1d4c0 -1 librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4) 2024-01-11T21:46:45.763 INFO:tasks.workunit.client.0.smithi178.stderr:(22) Invalid argument 2024-01-11T21:46:45.768 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:46:45.768 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test1 2024-01-11T21:46:45.927 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-01-11T21:46:45.924+0000 7f1e50b28700 0 -- 172.21.15.178:0/3239417891 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f1e28008740 msgr2=0x7f1e2800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:45.956 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:45.956+0000 7f1e50b28700 0 -- 172.21.15.178:0/3239417891 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f1e340372f0 msgr2=0x7f1e34039710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:45.967 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 100% complete...done. 2024-01-11T21:46:45.972 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd remove test1 2024-01-11T21:46:46.119 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-01-11T21:46:46.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/dev/urandom bs=1M count=1 2024-01-11T21:46:46.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd --image-format 2 import - test1 2024-01-11T21:46:46.199 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-01-11T21:46:46.199 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-01-11T21:46:46.199 INFO:tasks.workunit.client.0.smithi178.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737855 s, 14.2 MB/s 2024-01-11T21:46:46.257 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 100% complete...done. 2024-01-11T21:46:46.262 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd export test1 - 2024-01-11T21:46:46.262 INFO:tasks.workunit.client.0.smithi178.stderr:++ md5sum 2024-01-11T21:46:46.317 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:46:46.323 INFO:tasks.workunit.client.0.smithi178.stderr:+ md5sum='4e1de3e388bef2eb6bb3921f81134a27 -' 2024-01-11T21:46:46.323 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@snap1 2024-01-11T21:46:46.543 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:46:46.557 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect test1@snap1 2024-01-11T21:46:46.627 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@snap2 2024-01-11T21:46:47.551 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:46:47.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone test1@snap1 clone_v1 --rbd_default_clone_format=1 2024-01-11T21:46:47.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone test1@snap2 clone_v2 --rbd_default_clone_format=2 2024-01-11T21:46:47.787 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v1 2024-01-11T21:46:47.787 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'parent: rbd/test1@snap1' 2024-01-11T21:46:47.866 INFO:tasks.workunit.client.0.smithi178.stdout: parent: rbd/test1@snap1 2024-01-11T21:46:47.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v2 2024-01-11T21:46:47.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'parent: rbd/test1@snap2' 2024-01-11T21:46:47.938 INFO:tasks.workunit.client.0.smithi178.stdout: parent: rbd/test1@snap2 2024-01-11T21:46:47.938 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v2 2024-01-11T21:46:47.938 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'op_features: clone-child' 2024-01-11T21:46:48.014 INFO:tasks.workunit.client.0.smithi178.stdout: op_features: clone-child 2024-01-11T21:46:48.015 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd export clone_v1 - 2024-01-11T21:46:48.016 INFO:tasks.workunit.client.0.smithi178.stderr:++ md5sum 2024-01-11T21:46:48.095 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:46:48.102 INFO:tasks.workunit.client.0.smithi178.stderr:+ test '4e1de3e388bef2eb6bb3921f81134a27 -' = '4e1de3e388bef2eb6bb3921f81134a27 -' 2024-01-11T21:46:48.103 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd export clone_v2 - 2024-01-11T21:46:48.103 INFO:tasks.workunit.client.0.smithi178.stderr:++ md5sum 2024-01-11T21:46:48.182 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:46:48.188 INFO:tasks.workunit.client.0.smithi178.stderr:+ test '4e1de3e388bef2eb6bb3921f81134a27 -' = '4e1de3e388bef2eb6bb3921f81134a27 -' 2024-01-11T21:46:48.189 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd children test1@snap1 2024-01-11T21:46:48.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2024-01-11T21:46:48.263 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd children test1@snap2 2024-01-11T21:46:48.328 INFO:tasks.workunit.client.0.smithi178.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2024-01-11T21:46:48.328 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test1 rbd2/test2 2024-01-11T21:46:48.509 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:48.507+0000 7f62da4e2700 0 -- 172.21.15.178:0/704922916 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f62b4004bb0 msgr2=0x7f62b4005030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:49.523 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:49.519+0000 7f62da4e2700 0 -- 172.21.15.178:0/704922916 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f62bc037210 msgr2=0x7f62bc039610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:49.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v1 2024-01-11T21:46:49.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'parent: rbd2/test2@snap1' 2024-01-11T21:46:49.761 INFO:tasks.workunit.client.0.smithi178.stdout: parent: rbd2/test2@snap1 2024-01-11T21:46:49.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v2 2024-01-11T21:46:49.763 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'parent: rbd2/test2@snap2' 2024-01-11T21:46:49.864 INFO:tasks.workunit.client.0.smithi178.stdout: parent: rbd2/test2@snap2 2024-01-11T21:46:49.865 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v2 2024-01-11T21:46:49.865 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'op_features: clone-child' 2024-01-11T21:46:49.976 INFO:tasks.workunit.client.0.smithi178.stdout: op_features: clone-child 2024-01-11T21:46:49.977 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd children rbd2/test2@snap1 2024-01-11T21:46:50.101 INFO:tasks.workunit.client.0.smithi178.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2024-01-11T21:46:50.101 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd children rbd2/test2@snap2 2024-01-11T21:46:50.196 INFO:tasks.workunit.client.0.smithi178.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2024-01-11T21:46:50.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test1 2024-01-11T21:46:50.355 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:50.360 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd migration commit test1 2024-01-11T21:46:50.360 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 2024-01-11T21:46:50.507 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: the image has descendants: 2024-01-11T21:46:50.507 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v1 2024-01-11T21:46:50.507 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v2 2024-01-11T21:46:50.507 INFO:tasks.workunit.client.0.smithi178.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-01-11T21:46:50.507 INFO:tasks.workunit.client.0.smithi178.stderr:Ensure no descendant images are opened read-only and run again with force flag. 2024-01-11T21:46:50.512 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:46:50.512 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 --force 2024-01-11T21:46:50.654 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: the image has descendants: 2024-01-11T21:46:50.654 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v1 2024-01-11T21:46:50.654 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v2 2024-01-11T21:46:50.654 INFO:tasks.workunit.client.0.smithi178.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-01-11T21:46:50.654 INFO:tasks.workunit.client.0.smithi178.stderr:Proceeding anyway due to force flag set. 2024-01-11T21:46:50.682 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:50.679+0000 7fe8e74b5700 0 -- 172.21.15.178:0/3776257635 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55ea472bdf30 msgr2=0x55ea472c0310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:50.696 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:50.695+0000 7fe8e74b5700 0 -- 172.21.15.178:0/3776257635 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fe8c80372e0 msgr2=0x7fe8c80396e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:52.608 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 100% complete...done. 2024-01-11T21:46:52.613 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd export clone_v1 - 2024-01-11T21:46:52.613 INFO:tasks.workunit.client.0.smithi178.stderr:++ md5sum 2024-01-11T21:46:52.688 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:46:52.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ test '4e1de3e388bef2eb6bb3921f81134a27 -' = '4e1de3e388bef2eb6bb3921f81134a27 -' 2024-01-11T21:46:52.696 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd export clone_v2 - 2024-01-11T21:46:52.696 INFO:tasks.workunit.client.0.smithi178.stderr:++ md5sum 2024-01-11T21:46:52.773 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:46:52.779 INFO:tasks.workunit.client.0.smithi178.stderr:+ test '4e1de3e388bef2eb6bb3921f81134a27 -' = '4e1de3e388bef2eb6bb3921f81134a27 -' 2024-01-11T21:46:52.779 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare rbd2/test2 test1 2024-01-11T21:46:52.938 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:52.935+0000 7f69a66bc700 0 -- 172.21.15.178:0/1582336817 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f698c008740 msgr2=0x7f698c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:54.573 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:54.571+0000 7f69a66bc700 0 -- 172.21.15.178:0/1582336817 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f6988037360 msgr2=0x7f6988039780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:54.722 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v1 2024-01-11T21:46:54.722 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'parent: rbd/test1@snap1' 2024-01-11T21:46:54.829 INFO:tasks.workunit.client.0.smithi178.stdout: parent: rbd/test1@snap1 2024-01-11T21:46:54.830 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'parent: rbd/test1@snap2' 2024-01-11T21:46:54.830 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v2 2024-01-11T21:46:54.938 INFO:tasks.workunit.client.0.smithi178.stdout: parent: rbd/test1@snap2 2024-01-11T21:46:54.938 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info clone_v2 2024-01-11T21:46:54.938 INFO:tasks.workunit.client.0.smithi178.stderr:+ fgrep 'op_features: clone-child' 2024-01-11T21:46:55.049 INFO:tasks.workunit.client.0.smithi178.stdout: op_features: clone-child 2024-01-11T21:46:55.050 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd children test1@snap1 2024-01-11T21:46:55.160 INFO:tasks.workunit.client.0.smithi178.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2024-01-11T21:46:55.160 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd children test1@snap2 2024-01-11T21:46:55.255 INFO:tasks.workunit.client.0.smithi178.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2024-01-11T21:46:55.256 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test1 2024-01-11T21:46:55.434 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:55.431+0000 7fe6c71a8700 0 -- 172.21.15.178:0/1651590465 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fe6ac008740 msgr2=0x7fe6ac00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:55.437 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:46:55.441 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd migration commit test1 2024-01-11T21:46:55.441 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 2024-01-11T21:46:55.595 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: the image has descendants: 2024-01-11T21:46:55.596 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v1 2024-01-11T21:46:55.596 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v2 2024-01-11T21:46:55.596 INFO:tasks.workunit.client.0.smithi178.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-01-11T21:46:55.596 INFO:tasks.workunit.client.0.smithi178.stderr:Ensure no descendant images are opened read-only and run again with force flag. 2024-01-11T21:46:55.602 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:46:55.602 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration commit test1 --force 2024-01-11T21:46:55.757 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: the image has descendants: 2024-01-11T21:46:55.757 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v1 2024-01-11T21:46:55.757 INFO:tasks.workunit.client.0.smithi178.stderr: rbd/clone_v2 2024-01-11T21:46:55.758 INFO:tasks.workunit.client.0.smithi178.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-01-11T21:46:55.758 INFO:tasks.workunit.client.0.smithi178.stderr:Proceeding anyway due to force flag set. 2024-01-11T21:46:55.764 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:55.759+0000 7f0223708700 0 -- 172.21.15.178:0/1517496866 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f0208008740 msgr2=0x7f020800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:46:55.793 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:55.791+0000 7f0223708700 0 -- 172.21.15.178:0/1517496866 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f0204037270 msgr2=0x7f0204039690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:57.616 INFO:tasks.workunit.client.0.smithi178.stderr: Commit image migration: 100% complete...done. 2024-01-11T21:46:57.622 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd export clone_v1 - 2024-01-11T21:46:57.622 INFO:tasks.workunit.client.0.smithi178.stderr:++ md5sum 2024-01-11T21:46:57.697 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:46:57.704 INFO:tasks.workunit.client.0.smithi178.stderr:+ test '4e1de3e388bef2eb6bb3921f81134a27 -' = '4e1de3e388bef2eb6bb3921f81134a27 -' 2024-01-11T21:46:57.705 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd export clone_v2 - 2024-01-11T21:46:57.705 INFO:tasks.workunit.client.0.smithi178.stderr:++ md5sum 2024-01-11T21:46:57.785 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:46:57.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ test '4e1de3e388bef2eb6bb3921f81134a27 -' = '4e1de3e388bef2eb6bb3921f81134a27 -' 2024-01-11T21:46:57.793 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd remove clone_v1 2024-01-11T21:46:57.982 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:57.979+0000 7f1696df7700 0 -- 172.21.15.178:0/3211730248 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f1674008740 msgr2=0x7f167400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:57.990 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:57.995 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd remove clone_v2 2024-01-11T21:46:58.172 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:58.171+0000 7fae211b8700 0 -- 172.21.15.178:0/2381373334 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fadfc008740 msgr2=0x7fadfc00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:58.188 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:58.192 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect test1@snap1 2024-01-11T21:46:58.264 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge test1 2024-01-11T21:46:59.529 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:46:59.541 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:46:59.733 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:59.731+0000 7f47397af700 0 -- 172.21.15.178:0/2562230555 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55954bb6c020 msgr2=0x55954bc0c9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:46:59.738 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:46:59.743 INFO:tasks.workunit.client.0.smithi178.stderr:+ for format in 1 2 2024-01-11T21:46:59.743 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 1 test2 2024-01-11T21:46:59.769 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:46:59.783 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:46:59.779+0000 7f1cbe8434c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:46:59.794 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-01-11T21:47:00.126 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:00.183 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:00.189 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:00.189 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:00.198 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test2 2024-01-11T21:47:00.299 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-01-11T21:47:00.295+0000 7faac6373700 0 -- 172.21.15.178:0/3136164113 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7faaac008740 msgr2=0x7faaac00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:00.325 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 100% complete...done. 2024-01-11T21:47:00.328 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:00.379 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:00.384 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:00.384 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:00.392 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:00.466 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-01-11T21:47:00.470 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 1 test2 2024-01-11T21:47:00.494 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:47:00.505 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:00.503+0000 7f45138d74c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:00.517 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-01-11T21:47:00.855 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:00.923 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:00.931 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:00.931 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:00.940 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test2 2024-01-11T21:47:01.086 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete...2024-01-11T21:47:01.083+0000 7f4974b11700 0 -- 172.21.15.178:0/2776695618 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f49580064c0 msgr2=0x7f4958028160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:47:01.107 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:47:01.103+0000 7f4975312700 0 -- 172.21.15.178:0/2776695618 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f49540375c0 msgr2=0x7f49540399e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:01.129 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:47:01.132 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test2 2024-01-11T21:47:01.246 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-01-11T21:47:01.243+0000 7f6a32728700 0 -- 172.21.15.178:0/316871571 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55dd8261c610 msgr2=0x55dd827246f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:47:01.269 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 100% complete...done. 2024-01-11T21:47:01.274 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:01.326 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:01.330 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:01.331 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-01-11T21:47:01.338 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:01.422 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-01-11T21:47:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 1 test2 2024-01-11T21:47:01.452 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:47:01.462 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:01.459+0000 7f9382aee4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:01.475 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 --data-pool INVALID_DATA_POOL 2024-01-11T21:47:01.545 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:01.543+0000 7f41577dd4c0 -1 librbd::image::CreateRequest: 0x559a5a4253d0 validate_data_pool: data pool INVALID_DATA_POOL does not exist 2024-01-11T21:47:01.545 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:01.543+0000 7f41577dd4c0 -1 librbd::Migration: create_dst_image: header creation failed: (2) No such file or directory 2024-01-11T21:47:01.564 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: preparing migration failed: (2) No such file or directory 2024-01-11T21:47:01.569 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:47:01.569 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:01.626 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:01.631 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:01.631 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-01-11T21:47:01.639 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:01.718 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-01-11T21:47:01.721 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 1 test2 2024-01-11T21:47:01.747 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:47:01.757 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:01.755+0000 7fe0eec304c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:01.769 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 rbd2/test2 2024-01-11T21:47:01.893 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/test2 2024-01-11T21:47:01.962 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:01.970 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:01.970 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:01.977 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test2 2024-01-11T21:47:02.098 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2024-01-11T21:47:02.102 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:02.148 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:02.155 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:02.155 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-01-11T21:47:02.162 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:02.238 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-01-11T21:47:02.241 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 1 test2 2024-01-11T21:47:02.266 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image format 1 is deprecated 2024-01-11T21:47:02.275 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:02.271+0000 7f73573914c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:02.287 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 rbd2/test2 2024-01-11T21:47:02.399 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort rbd2/test2 2024-01-11T21:47:02.516 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-01-11T21:47:02.515+0000 7efcdbbb4700 0 -- 172.21.15.178:0/3100848808 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7efcc0008740 msgr2=0x7efcc000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:02.529 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 100% complete...done. 2024-01-11T21:47:02.533 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:02.583 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:02.588 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:02.588 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:02.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:02.671 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-01-11T21:47:02.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ test 1 = 1 2024-01-11T21:47:02.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ continue 2024-01-11T21:47:02.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ for format in 1 2 2024-01-11T21:47:02.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 2 test2 2024-01-11T21:47:02.731 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-01-11T21:47:02.924 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:02.923+0000 7efca41d1700 0 -- 172.21.15.178:0/593266303 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7efc840375c0 msgr2=0x7efc840399e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:02.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:03.032 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:03.041 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:03.041 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 83.3352 bytes/sec: 83 KiB/s 2024-01-11T21:47:03.055 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test2 2024-01-11T21:47:03.265 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-01-11T21:47:03.263+0000 7f58fc8e8700 0 -- 172.21.15.178:0/709841089 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f58e0008740 msgr2=0x7f58e000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:03.271 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 100% complete...done. 2024-01-11T21:47:03.275 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:03.341 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:03.345 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:03.345 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-01-11T21:47:03.359 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:03.549 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-01-11T21:47:03.547+0000 7fa237b27700 0 -- 172.21.15.178:0/2149728574 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fa214008740 msgr2=0x7fa21400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:47:03.567 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:03.572 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 2 test2 2024-01-11T21:47:03.648 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-01-11T21:47:03.849 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:03.847+0000 7fbcc8e05700 0 -- 172.21.15.178:0/607900726 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x561326369fc0 msgr2=0x561326473c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:03.861 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:03.949 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:03.958 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:03.958 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:03.972 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration execute test2 2024-01-11T21:47:04.135 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-01-11T21:47:04.131+0000 7f0320cec700 0 -- 172.21.15.178:0/463128103 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5596f6e02580 msgr2=0x5596f6e04960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:04.165 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:04.163+0000 7f0320cec700 0 -- 172.21.15.178:0/463128103 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f03040375c0 msgr2=0x7f03040399e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:04.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:47:04.163+0000 7fdf82bba700 -1 reset not still connected to 0x55912792e340 2024-01-11T21:47:04.171 INFO:tasks.workunit.client.0.smithi178.stderr: Image migration: 100% complete...done. 2024-01-11T21:47:04.175 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test2 2024-01-11T21:47:04.368 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-01-11T21:47:04.367+0000 7ff46cbee700 0 -- 172.21.15.178:0/2257023639 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55a0b4a5f920 msgr2=0x55a0b4a61d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:04.374 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 100% complete...done. 2024-01-11T21:47:04.379 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:04.449 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:04.453 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:04.454 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-01-11T21:47:04.471 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:04.673 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-01-11T21:47:04.671+0000 7fbed2276700 0 -- 172.21.15.178:0/1058058990 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5612179bb4a0 msgr2=0x561217a778f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:04.688 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:04.692 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 2 test2 2024-01-11T21:47:04.759 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 --data-pool INVALID_DATA_POOL 2024-01-11T21:47:04.871 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:04.867+0000 7fa76c5834c0 -1 librbd::image::CreateRequest: 0x5585c6664d30 validate_data_pool: data pool INVALID_DATA_POOL does not exist 2024-01-11T21:47:04.871 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:04.867+0000 7fa76c5834c0 -1 librbd::Migration: create_dst_image: header creation failed: (2) No such file or directory 2024-01-11T21:47:04.915 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: preparing migration failed: (2) No such file or directory 2024-01-11T21:47:04.919 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:47:04.919 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:04.990 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:04.999 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:04.999 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:05.018 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:05.416 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-01-11T21:47:05.415+0000 7f01e35ba700 0 -- 172.21.15.178:0/1114750040 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f01c8008740 msgr2=0x7f01c800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:05.443 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:05.448 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 2 test2 2024-01-11T21:47:05.517 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 rbd2/test2 2024-01-11T21:47:05.687 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:05.683+0000 7f0b80fe1700 0 -- 172.21.15.178:0/495995516 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55c90842c500 msgr2=0x55c9083deb70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:05.697 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/test2 2024-01-11T21:47:05.775 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:05.785 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:05.785 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 83.3352 bytes/sec: 83 KiB/s 2024-01-11T21:47:05.795 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test2 2024-01-11T21:47:05.973 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2024-01-11T21:47:05.977 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:06.040 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:06.044 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:06.044 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-01-11T21:47:06.060 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:06.248 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-01-11T21:47:06.247+0000 7f02a6534700 0 -- 172.21.15.178:0/1137724509 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55c139ae0f20 msgr2=0x55c139b81ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:47:06.258 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:06.262 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 2 test2 2024-01-11T21:47:06.334 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 rbd2/test2 2024-01-11T21:47:06.510 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:06.507+0000 7f9c53cd9700 0 -- 172.21.15.178:0/1232840938 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f9c38008740 msgr2=0x7f9c3800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:06.520 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort rbd2/test2 2024-01-11T21:47:06.692 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2024-01-11T21:47:06.696 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:06.766 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:06.774 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:06.774 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:06.790 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:07.004 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-01-11T21:47:07.003+0000 7f1b01495700 0 -- 172.21.15.178:0/1689570788 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55b073c4f080 msgr2=0x55b073c51460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:07.008 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:07.013 INFO:tasks.workunit.client.0.smithi178.stderr:+ test 2 = 1 2024-01-11T21:47:07.013 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 128M --image-format 2 test2 2024-01-11T21:47:07.076 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration prepare test2 rbd2/ns1/test3 2024-01-11T21:47:07.246 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:07.243+0000 7f343b32e700 0 -- 172.21.15.178:0/1946885033 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55fde7b19c60 msgr2=0x55fde7b1c040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:07.258 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/ns1/test3 2024-01-11T21:47:07.340 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:07.348 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:07.348 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-01-11T21:47:07.359 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd migration abort test2 2024-01-11T21:47:07.528 INFO:tasks.workunit.client.0.smithi178.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2024-01-11T21:47:07.534 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-01-11T21:47:07.597 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-01-11T21:47:07.600 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:47:07.600 INFO:tasks.workunit.client.0.smithi178.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-01-11T21:47:07.617 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:47:07.824 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-01-11T21:47:07.823+0000 7fdb232aa700 0 -- 172.21.15.178:0/3451580015 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x559707252080 msgr2=0x559707254460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:07.831 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:07.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:47:07.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:07.968 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:08.095 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:08.225 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:08.358 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:08.487 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:08.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:08.746 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:08.871 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:09.002 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:09.333 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:09.460 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:09.594 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:09.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:09.863 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:09.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:10.117 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:10.241 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:10.367 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-01-11T21:47:11.636 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd2' does not exist 2024-01-11T21:47:11.644 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_config 2024-01-11T21:47:11.644 INFO:tasks.workunit.client.0.smithi178.stdout:testing config... 2024-01-11T21:47:11.645 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing config...' 2024-01-11T21:47:11.645 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:47:11.645 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:11.769 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:11.891 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.019 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.153 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.287 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.551 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.678 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.812 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:12.942 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.329 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.459 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.582 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.713 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.846 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:13.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config global set osd rbd_cache true 2024-01-11T21:47:13.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global set osd rbd_cache true 2024-01-11T21:47:14.004 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: invalid config entity: osd (must be global, client or client.) 2024-01-11T21:47:14.006 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:14.006 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config global set global debug_ms 10 2024-01-11T21:47:14.006 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global set global debug_ms 10 2024-01-11T21:47:14.036 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: not rbd option: debug_ms 2024-01-11T21:47:14.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:14.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config global set global rbd_UNKNOWN false 2024-01-11T21:47:14.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global set global rbd_UNKNOWN false 2024-01-11T21:47:14.069 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: invalid config key: rbd_UNKNOWN 2024-01-11T21:47:14.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:14.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config global set global rbd_cache INVALID 2024-01-11T21:47:14.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global set global rbd_cache INVALID 2024-01-11T21:47:14.111 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: error setting rbd_cache: error parsing value: Expected option value to be integer, got 'INVALID' 2024-01-11T21:47:14.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:14.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global set global rbd_cache false 2024-01-11T21:47:14.161 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global set client rbd_cache true 2024-01-11T21:47:14.205 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global set client.123 rbd_cache false 2024-01-11T21:47:14.254 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global get global rbd_cache 2024-01-11T21:47:14.254 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^false$' 2024-01-11T21:47:14.293 INFO:tasks.workunit.client.0.smithi178.stdout:false 2024-01-11T21:47:14.295 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global get client rbd_cache 2024-01-11T21:47:14.295 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^true$' 2024-01-11T21:47:14.338 INFO:tasks.workunit.client.0.smithi178.stdout:true 2024-01-11T21:47:14.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global get client.123 rbd_cache 2024-01-11T21:47:14.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^false$' 2024-01-11T21:47:14.382 INFO:tasks.workunit.client.0.smithi178.stdout:false 2024-01-11T21:47:14.383 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config global get client.UNKNOWN rbd_cache 2024-01-11T21:47:14.383 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global get client.UNKNOWN rbd_cache 2024-01-11T21:47:14.422 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: rbd_cache is not set 2024-01-11T21:47:14.426 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:14.427 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global list global 2024-01-11T21:47:14.427 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * false * global *$' 2024-01-11T21:47:14.467 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache false global 2024-01-11T21:47:14.468 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global list client 2024-01-11T21:47:14.468 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * true * client *$' 2024-01-11T21:47:14.508 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache true client 2024-01-11T21:47:14.508 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global list client.123 2024-01-11T21:47:14.509 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * false * client.123 *$' 2024-01-11T21:47:14.549 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache false client.123 2024-01-11T21:47:14.550 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global list client.UNKNOWN 2024-01-11T21:47:14.550 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * true * client *$' 2024-01-11T21:47:14.589 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache true client 2024-01-11T21:47:14.589 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global rm client rbd_cache 2024-01-11T21:47:14.635 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config global get client rbd_cache 2024-01-11T21:47:14.635 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global get client rbd_cache 2024-01-11T21:47:14.671 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: rbd_cache is not set 2024-01-11T21:47:14.674 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:14.674 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global list client 2024-01-11T21:47:14.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * false * global *$' 2024-01-11T21:47:14.712 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache false global 2024-01-11T21:47:14.712 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global rm client.123 rbd_cache 2024-01-11T21:47:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config global rm global rbd_cache 2024-01-11T21:47:14.808 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config pool set rbd rbd_cache true 2024-01-11T21:47:14.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config pool list rbd 2024-01-11T21:47:14.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * true * pool *$' 2024-01-11T21:47:14.906 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache true pool 2024-01-11T21:47:14.907 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config pool get rbd rbd_cache 2024-01-11T21:47:14.907 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^true$' 2024-01-11T21:47:14.949 INFO:tasks.workunit.client.0.smithi178.stdout:true 2024-01-11T21:47:14.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 1 test1 2024-01-11T21:47:14.985 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:14.983+0000 7f83d49514c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:14.999 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config image list rbd/test1 2024-01-11T21:47:14.999 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * true * pool *$' 2024-01-11T21:47:15.062 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache true pool 2024-01-11T21:47:15.062 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config image set rbd/test1 rbd_cache false 2024-01-11T21:47:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config image list rbd/test1 2024-01-11T21:47:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * false * image *$' 2024-01-11T21:47:15.196 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache false image 2024-01-11T21:47:15.196 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config image get rbd/test1 rbd_cache 2024-01-11T21:47:15.196 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^false$' 2024-01-11T21:47:15.255 INFO:tasks.workunit.client.0.smithi178.stdout:false 2024-01-11T21:47:15.255 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config image remove rbd/test1 rbd_cache 2024-01-11T21:47:15.324 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config image get rbd/test1 rbd_cache 2024-01-11T21:47:15.324 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config image get rbd/test1 rbd_cache 2024-01-11T21:47:15.372 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: rbd_cache is not set 2024-01-11T21:47:15.380 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:15.380 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config image list rbd/test1 2024-01-11T21:47:15.381 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * true * pool *$' 2024-01-11T21:47:15.441 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache true pool 2024-01-11T21:47:15.441 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config pool remove rbd rbd_cache 2024-01-11T21:47:15.498 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd config pool get rbd rbd_cache 2024-01-11T21:47:15.498 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config pool get rbd rbd_cache 2024-01-11T21:47:15.541 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: rbd_cache is not set 2024-01-11T21:47:15.546 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:47:15.547 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd config pool list rbd 2024-01-11T21:47:15.547 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^rbd_cache * true * config *$' 2024-01-11T21:47:15.585 INFO:tasks.workunit.client.0.smithi178.stdout:rbd_cache true config 2024-01-11T21:47:15.586 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:47:15.648 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:15.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ RBD_CREATE_ARGS= 2024-01-11T21:47:15.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_others 2024-01-11T21:47:15.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing import, export, resize, and snapshots...' 2024-01-11T21:47:15.652 INFO:tasks.workunit.client.0.smithi178.stdout:testing import, export, resize, and snapshots... 2024-01-11T21:47:15.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ TMP_FILES='/tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1' 2024-01-11T21:47:15.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:47:15.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:15.783 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:15.909 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:16.243 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:16.371 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:16.496 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:16.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:16.767 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:16.889 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.020 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.155 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.275 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.402 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.532 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.658 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.788 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:17.914 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:18.040 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:18.177 INFO:tasks.workunit.client.0.smithi178.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-01-11T21:47:18.178 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/sh of=/tmp/img1 bs=1k count=1 seek=10 2024-01-11T21:47:18.180 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-01-11T21:47:18.180 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-01-11T21:47:18.180 INFO:tasks.workunit.client.0.smithi178.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000175782 s, 5.8 MB/s 2024-01-11T21:47:18.180 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/dd of=/tmp/img1 bs=1k count=10 seek=100 2024-01-11T21:47:18.181 INFO:tasks.workunit.client.0.smithi178.stderr:10+0 records in 2024-01-11T21:47:18.181 INFO:tasks.workunit.client.0.smithi178.stderr:10+0 records out 2024-01-11T21:47:18.182 INFO:tasks.workunit.client.0.smithi178.stderr:10240 bytes (10 kB, 10 KiB) copied, 0.000189146 s, 54.1 MB/s 2024-01-11T21:47:18.182 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/rm of=/tmp/img1 bs=1k count=100 seek=1000 2024-01-11T21:47:18.183 INFO:tasks.workunit.client.0.smithi178.stderr:70+1 records in 2024-01-11T21:47:18.183 INFO:tasks.workunit.client.0.smithi178.stderr:70+1 records out 2024-01-11T21:47:18.183 INFO:tasks.workunit.client.0.smithi178.stderr:72056 bytes (72 kB, 70 KiB) copied, 0.000551363 s, 131 MB/s 2024-01-11T21:47:18.183 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/ls of=/tmp/img1 bs=1k seek=10000 2024-01-11T21:47:18.186 INFO:tasks.workunit.client.0.smithi178.stderr:138+1 records in 2024-01-11T21:47:18.186 INFO:tasks.workunit.client.0.smithi178.stderr:138+1 records out 2024-01-11T21:47:18.186 INFO:tasks.workunit.client.0.smithi178.stderr:142144 bytes (142 kB, 139 KiB) copied, 0.000986199 s, 144 MB/s 2024-01-11T21:47:18.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/ln of=/tmp/img1 bs=1k seek=100000 2024-01-11T21:47:18.187 INFO:tasks.workunit.client.0.smithi178.stderr:74+1 records in 2024-01-11T21:47:18.187 INFO:tasks.workunit.client.0.smithi178.stderr:74+1 records out 2024-01-11T21:47:18.187 INFO:tasks.workunit.client.0.smithi178.stderr:76160 bytes (76 kB, 74 KiB) copied, 0.000336073 s, 227 MB/s 2024-01-11T21:47:18.187 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import /tmp/img1 testimg1 2024-01-11T21:47:18.229 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:18.227+0000 7f2c0c9d34c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:18.335 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 4% complete... Importing image: 8% complete... Importing image: 12% complete... Importing image: 16% complete... Importing image: 20% complete... Importing image: 24% complete... Importing image: 28% complete... Importing image: 32% complete... Importing image: 36% complete... Importing image: 40% complete... Importing image: 45% complete... Importing image: 49% complete... Importing image: 53% complete... Importing image: 57% complete... Importing image: 61% complete... Importing image: 65% complete... Importing image: 69% complete... Importing image: 73% complete... Importing image: 77% complete... Importing image: 81% complete... Importing image: 85% complete... Importing image: 90% complete... Importing image: 94% complete... Importing image: 98% complete... Importing image: 100% complete...done. 2024-01-11T21:47:18.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd resize testimg1 --size=256 --allow-shrink 2024-01-11T21:47:18.387 INFO:tasks.workunit.client.0.smithi178.stderr: Resizing image: 100% complete...done. 2024-01-11T21:47:18.395 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img2 2024-01-11T21:47:18.527 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:18.532 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1 --snap=snap1 2024-01-11T21:47:19.552 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:47:19.560 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd resize testimg1 --size=128 2024-01-11T21:47:19.610 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: shrinking an image is only allowed with the --allow-shrink flag 2024-01-11T21:47:19.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:47:19.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd resize testimg1 --size=128 --allow-shrink 2024-01-11T21:47:19.699 INFO:tasks.workunit.client.0.smithi178.stderr: Resizing image: 50% complete... Resizing image: 51% complete... Resizing image: 53% complete... Resizing image: 54% complete... Resizing image: 56% complete... Resizing image: 57% complete... Resizing image: 59% complete... Resizing image: 60% complete... Resizing image: 62% complete... Resizing image: 64% complete... Resizing image: 65% complete... Resizing image: 67% complete... Resizing image: 68% complete... Resizing image: 70% complete... Resizing image: 71% complete... Resizing image: 73% complete... Resizing image: 75% complete... Resizing image: 76% complete... Resizing image: 78% complete... Resizing image: 79% complete... Resizing image: 81% complete... Resizing image: 82% complete... Resizing image: 84% complete... Resizing image: 85% complete... Resizing image: 87% complete... Resizing image: 89% complete... Resizing image: 90% complete... Resizing image: 92% complete... Resizing image: 93% complete... Resizing image: 95% complete... Resizing image: 96% complete... Resizing image: 98% complete... Resizing image: 100% complete...done. 2024-01-11T21:47:19.705 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img3 2024-01-11T21:47:19.792 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:19.796 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg1 2024-01-11T21:47:19.796 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:19.842 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg1 2024-01-11T21:47:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:19.895 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:19.895 INFO:tasks.workunit.client.0.smithi178.stderr:+ rm -rf /tmp/diff-testimg1-1 /tmp/diff-testimg1-2 2024-01-11T21:47:19.895 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export-diff testimg1 --snap=snap1 /tmp/diff-testimg1-1 2024-01-11T21:47:20.056 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 37% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:20.060 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export-diff testimg1 --from-snap=snap1 /tmp/diff-testimg1-2 2024-01-11T21:47:20.144 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:47:20.148 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --size=1 testimg-diff1 2024-01-11T21:47:20.182 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:20.179+0000 7fb4c48ce4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:20.194 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-1 testimg-diff1 2024-01-11T21:47:20.558 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image diff: 22% complete... Importing image diff: 63% complete... Importing image diff: 99% complete... Importing image diff: 100% complete...done. 2024-01-11T21:47:20.567 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-2 testimg-diff1 2024-01-11T21:47:20.641 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image diff: 68% complete... Importing image diff: 96% complete... Importing image diff: 100% complete...done. 2024-01-11T21:47:20.647 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg1 2024-01-11T21:47:20.647 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:20.692 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:20.693 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg1 2024-01-11T21:47:20.693 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:20.745 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:20.745 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff1 2024-01-11T21:47:20.746 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:20.797 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:20.798 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-01-11T21:47:20.798 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:20.853 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:20.853 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg1 --snap=snap1 testimg2 2024-01-11T21:47:20.984 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 1% complete... Image copy: 3% complete... Image copy: 4% complete... Image copy: 6% complete... Image copy: 7% complete... Image copy: 9% complete... Image copy: 10% complete... Image copy: 12% complete... Image copy: 14% complete... Image copy: 15% complete... Image copy: 17% complete... Image copy: 18% complete... Image copy: 20% complete... Image copy: 21% complete... Image copy: 23% complete... Image copy: 25% complete... Image copy: 26% complete... Image copy: 28% complete... Image copy: 29% complete... Image copy: 31% complete... Image copy: 32% complete... Image copy: 34% complete... Image copy: 35% complete... Image copy: 37% complete... Image copy: 39% complete... Image copy: 40% complete... Image copy: 42% complete... Image copy: 43% complete... Image copy: 45% complete... Image copy: 46% complete... Image copy: 48% complete... Image copy: 50% complete... Image copy: 51% complete... Image copy: 53% complete... Image copy: 54% complete... Image copy: 56% complete... Image copy: 57% complete... Image copy: 59% complete... Image copy: 60% complete... Image copy: 62% complete... Image copy: 64% complete... Image copy: 65% complete... Image copy: 67% complete... Image copy: 68% complete... Image copy: 70% complete... Image copy: 71% complete... Image copy: 73% complete... Image copy: 75% complete... Image copy: 76% complete... Image copy: 78% complete... Image copy: 79% complete... Image copy: 81% complete... Image copy: 82% complete... Image copy: 84% complete... Image copy: 85% complete... Image copy: 87% complete... Image copy: 89% complete... Image copy: 90% complete... Image copy: 92% complete... Image copy: 93% complete... Image copy: 95% complete... Image copy: 96% complete... Image copy: 98% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:20.988 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg1 testimg3 2024-01-11T21:47:21.105 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:21.109 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg-diff1 --sparse-size 768K --snap=snap1 testimg-diff2 2024-01-11T21:47:21.247 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 1% complete... Image copy: 3% complete... Image copy: 4% complete... Image copy: 6% complete... Image copy: 7% complete... Image copy: 9% complete... Image copy: 10% complete... Image copy: 12% complete... Image copy: 14% complete... Image copy: 15% complete... Image copy: 17% complete... Image copy: 18% complete... Image copy: 20% complete... Image copy: 21% complete... Image copy: 23% complete... Image copy: 25% complete... Image copy: 26% complete... Image copy: 28% complete... Image copy: 29% complete... Image copy: 31% complete... Image copy: 32% complete... Image copy: 34% complete... Image copy: 35% complete... Image copy: 37% complete... Image copy: 39% complete... Image copy: 40% complete... Image copy: 42% complete... Image copy: 43% complete... Image copy: 45% complete... Image copy: 46% complete... Image copy: 48% complete... Image copy: 50% complete... Image copy: 51% complete... Image copy: 53% complete... Image copy: 54% complete... Image copy: 56% complete... Image copy: 57% complete... Image copy: 59% complete... Image copy: 60% complete... Image copy: 62% complete... Image copy: 64% complete... Image copy: 65% complete... Image copy: 67% complete... Image copy: 68% complete... Image copy: 70% complete... Image copy: 71% complete... Image copy: 73% complete... Image copy: 75% complete... Image copy: 76% complete... Image copy: 78% complete... Image copy: 79% complete... Image copy: 81% complete... Image copy: 82% complete... Image copy: 84% complete... Image copy: 85% complete... Image copy: 87% complete... Image copy: 89% complete... Image copy: 90% complete... Image copy: 92% complete... Image copy: 93% complete... Image copy: 95% complete... Image copy: 96% complete... Image copy: 98% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:21.251 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg-diff1 --sparse-size 768K testimg-diff3 2024-01-11T21:47:21.364 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:21.368 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg2 2024-01-11T21:47:21.369 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:21.415 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:21.416 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg3 2024-01-11T21:47:21.416 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:21.461 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff2 2024-01-11T21:47:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:21.518 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:21.519 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff3 2024-01-11T21:47:21.519 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:21.752 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:21.751+0000 7f8275cd2700 0 --2- 172.21.15.178:0/3291630913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd632eae50 0x55cd632e2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:47:21.781 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:21.782 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep copy testimg1 testimg4 2024-01-11T21:47:22.652 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-01-11T21:47:22.656 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep copy testimg1 --snap=snap1 testimg5 2024-01-11T21:47:23.642 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-01-11T21:47:23.647 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg4 2024-01-11T21:47:23.647 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:23.697 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:23.697 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg5 2024-01-11T21:47:23.698 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:23.748 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:23.749 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg4 2024-01-11T21:47:23.749 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:47:23.749 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:47:23.750 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:47:23.801 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:47:23.801 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg4 2024-01-11T21:47:23.801 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '.*snap1.*' 2024-01-11T21:47:23.861 INFO:tasks.workunit.client.0.smithi178.stdout: 18 snap1 256 MiB Thu Jan 11 21:47:22 2024 2024-01-11T21:47:23.861 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img1.new 2024-01-11T21:47:23.961 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:23.965 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg2 /tmp/img2.new 2024-01-11T21:47:24.116 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:24.120 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg3 /tmp/img3.new 2024-01-11T21:47:24.220 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:24.225 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.new 2024-01-11T21:47:24.318 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:24.321 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff2 /tmp/img-diff2.new 2024-01-11T21:47:24.481 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:24.486 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff3 /tmp/img-diff3.new 2024-01-11T21:47:24.584 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:24.588 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img2.new 2024-01-11T21:47:24.819 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img3 /tmp/img3.new 2024-01-11T21:47:24.937 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img-diff2.new 2024-01-11T21:47:25.136 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img3 /tmp/img-diff3.new 2024-01-11T21:47:25.240 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rollback --snap=snap1 testimg1 2024-01-11T21:47:25.365 INFO:tasks.workunit.client.0.smithi178.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-01-11T21:47:25.372 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rollback --snap=snap1 testimg-diff1 2024-01-11T21:47:25.520 INFO:tasks.workunit.client.0.smithi178.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-01-11T21:47:25.525 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg1 2024-01-11T21:47:25.526 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:25.568 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:25.569 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff1 2024-01-11T21:47:25.569 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:25.618 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:25.618 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img1.snap1 2024-01-11T21:47:25.751 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:25.754 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.snap1 2024-01-11T21:47:25.885 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:25.890 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img1.snap1 2024-01-11T21:47:26.080 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img-diff1.snap1 2024-01-11T21:47:26.273 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm testimg2 2024-01-11T21:47:26.422 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete...2024-01-11T21:47:26.419+0000 7f76b39d5700 0 -- 172.21.15.178:0/1448853166 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f7694036c40 msgr2=0x7f7694039020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:26.438 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-01-11T21:47:26.435+0000 7f76b39d5700 0 -- 172.21.15.178:0/1448853166 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x561be4624020 msgr2=0x7f769403bf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:26.446 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:26.450 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm testimg3 2024-01-11T21:47:26.605 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-01-11T21:47:26.610 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create testimg2 -s 0 2024-01-11T21:47:26.645 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:26.643+0000 7ff4df2014c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:26.658 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd cp testimg2 testimg3 2024-01-11T21:47:26.740 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 100% complete...done. 2024-01-11T21:47:26.744 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep cp testimg2 testimg6 2024-01-11T21:47:26.834 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 100% complete...done. 2024-01-11T21:47:26.839 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm --snap=snap1 testimg1 2024-01-11T21:47:27.734 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:47:27.740 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm --snap=snap1 testimg-diff1 2024-01-11T21:47:28.750 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:47:28.757 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg1 2024-01-11T21:47:28.757 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-01-11T21:47:28.805 INFO:tasks.workunit.client.0.smithi178.stdout:error setting snapshot context: (2) No such file or directory 2024-01-11T21:47:28.806 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-01-11T21:47:28.806 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-01-11T21:47:28.857 INFO:tasks.workunit.client.0.smithi178.stdout:error setting snapshot context: (2) No such file or directory 2024-01-11T21:47:28.858 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd sparsify testimg1 2024-01-11T21:47:29.017 INFO:tasks.workunit.client.0.smithi178.stderr: Image sparsify: 1% complete... Image sparsify: 3% complete... Image sparsify: 4% complete... Image sparsify: 6% complete... Image sparsify: 7% complete... Image sparsify: 9% complete... Image sparsify: 10% complete... Image sparsify: 12% complete... Image sparsify: 14% complete... Image sparsify: 15% complete... Image sparsify: 17% complete... Image sparsify: 18% complete... Image sparsify: 20% complete... Image sparsify: 21% complete... Image sparsify: 23% complete... Image sparsify: 25% complete... Image sparsify: 26% complete... Image sparsify: 28% complete... Image sparsify: 29% complete... Image sparsify: 31% complete... Image sparsify: 32% complete... Image sparsify: 34% complete... Image sparsify: 35% complete... Image sparsify: 37% complete... Image sparsify: 39% complete... Image sparsify: 40% complete... Image sparsify: 42% complete... Image sparsify: 43% complete... Image sparsify: 45% complete... Image sparsify: 46% complete... Image sparsify: 48% complete... Image sparsify: 50% complete... Image sparsify: 51% complete... Image sparsify: 53% complete... Image sparsify: 54% complete... Image sparsify: 56% complete... Image sparsify: 57% complete... Image sparsify: 59% complete... Image sparsify: 60% complete... Image sparsify: 62% complete... Image sparsify: 64% complete... Image sparsify: 65% complete... Image sparsify: 67% complete... Image sparsify: 68% complete... Image sparsify: 70% complete... Image sparsify: 71% complete... Image sparsify: 73% complete... Image sparsify: 75% complete... Image sparsify: 76% complete... Image sparsify: 78% complete... Image sparsify: 79% complete... Image sparsify: 81% complete... Image sparsify: 82% complete... Image sparsify: 84% complete... Image sparsify: 85% complete... Image sparsify: 87% complete... Image sparsify: 89% complete... Image sparsify: 90% complete... Image sparsify: 92% complete... Image sparsify: 93% complete... Image sparsify: 95% complete... Image sparsify: 96% complete... Image sparsify: 98% complete... Image sparsify: 100% complete...done. 2024-01-11T21:47:29.027 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:47:29.027 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:29.179 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:29.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:29.516 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:30.709 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:31.731 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:31.954 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:32.122 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:32.369 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:32.604 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:32.731 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:32.849 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:32.984 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:33.117 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:33.249 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:33.379 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:33.511 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:33.646 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:33.779 INFO:tasks.workunit.client.0.smithi178.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-01-11T21:47:33.883 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_locking 2024-01-11T21:47:33.883 INFO:tasks.workunit.client.0.smithi178.stdout:testing locking... 2024-01-11T21:47:33.883 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing locking...' 2024-01-11T21:47:33.883 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:47:33.884 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.011 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.134 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.269 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.404 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.540 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.663 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.801 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:34.930 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:35.057 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:35.190 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:35.323 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:35.661 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:35.788 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:35.922 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:36.057 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:36.189 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:36.325 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:36.462 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create -s 1 test1 2024-01-11T21:47:36.504 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:36.502+0000 7f316de004c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:36.519 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:47:36.519 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:47:36.519 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:47:36.571 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:47:36.571 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id 2024-01-11T21:47:36.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:47:36.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 1 ' 2024-01-11T21:47:36.688 INFO:tasks.workunit.client.0.smithi178.stdout:There is 1 exclusive lock on this image. 2024-01-11T21:47:36.689 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd lock list test1 2024-01-11T21:47:36.689 INFO:tasks.workunit.client.0.smithi178.stderr:++ tail -n 1 2024-01-11T21:47:36.689 INFO:tasks.workunit.client.0.smithi178.stderr:++ awk '{print $1;}' 2024-01-11T21:47:36.744 INFO:tasks.workunit.client.0.smithi178.stderr:+ LOCKER=client.7778 2024-01-11T21:47:36.744 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock remove test1 id client.7778 2024-01-11T21:47:37.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:47:37.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:47:37.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:47:37.609 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:47:37.609 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id --shared tag 2024-01-11T21:47:37.678 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 1 ' 2024-01-11T21:47:37.678 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:47:37.726 INFO:tasks.workunit.client.0.smithi178.stdout:There is 1 shared lock on this image. 2024-01-11T21:47:37.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id --shared tag 2024-01-11T21:47:37.791 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:47:37.791 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 2 ' 2024-01-11T21:47:38.021 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:38.018+0000 7fc0e0a03700 0 --2- 172.21.15.178:0/2691656524 >> v2:172.21.15.178:3300/0 conn(0x55dd18593cc0 0x55dd18594090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:47:38.048 INFO:tasks.workunit.client.0.smithi178.stdout:There are 2 shared locks on this image. 2024-01-11T21:47:38.048 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id2 --shared tag 2024-01-11T21:47:38.120 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:47:38.120 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 3 ' 2024-01-11T21:47:38.173 INFO:tasks.workunit.client.0.smithi178.stdout:There are 3 shared locks on this image. 2024-01-11T21:47:38.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:47:38.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ tail -n 1 2024-01-11T21:47:38.175 INFO:tasks.workunit.client.0.smithi178.stderr:+ awk '{print $2, $1;}' 2024-01-11T21:47:38.175 INFO:tasks.workunit.client.0.smithi178.stderr:+ xargs rbd lock remove test1 2024-01-11T21:47:38.576 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info test1 2024-01-11T21:47:38.576 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -qE 'features:.*exclusive' 2024-01-11T21:47:38.629 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:47:38.693 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:38.698 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_thick_provision 2024-01-11T21:47:38.698 INFO:tasks.workunit.client.0.smithi178.stdout:testing thick provision... 2024-01-11T21:47:38.698 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing thick provision...' 2024-01-11T21:47:38.698 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:47:38.699 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.170 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.307 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.440 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.574 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.704 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.838 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:39.969 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:40.093 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:40.216 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:40.353 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:40.487 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:40.623 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:40.753 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:40.886 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:41.019 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:41.155 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:41.280 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --thick-provision -s 64M test1 2024-01-11T21:47:41.321 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:41.318+0000 7f2fdca474c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:41.531 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 6% complete... Thick provisioning: 12% complete... Thick provisioning: 18% complete... Thick provisioning: 25% complete... Thick provisioning: 31% complete... Thick provisioning: 37% complete... Thick provisioning: 43% complete... Thick provisioning: 50% complete... Thick provisioning: 56% complete... Thick provisioning: 62% complete... Thick provisioning: 68% complete... Thick provisioning: 75% complete... Thick provisioning: 81% complete... Thick provisioning: 87% complete... Thick provisioning: 93% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2024-01-11T21:47:41.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ count=0 2024-01-11T21:47:41.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret= 2024-01-11T21:47:41.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 -lt 10 ']' 2024-01-11T21:47:41.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:47:41.538 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:47:41.539 INFO:tasks.workunit.client.0.smithi178.stderr:+ tr -s ' ' 2024-01-11T21:47:41.539 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^64 MiB' 2024-01-11T21:47:41.539 INFO:tasks.workunit.client.0.smithi178.stderr:+ cut -d ' ' -f 4-5 2024-01-11T21:47:41.579 INFO:tasks.workunit.client.0.smithi178.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2024-01-11T21:47:41.586 INFO:tasks.workunit.client.0.smithi178.stdout:64 MiB 2024-01-11T21:47:41.587 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret=0 2024-01-11T21:47:41.587 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 = 0 ']' 2024-01-11T21:47:41.587 INFO:tasks.workunit.client.0.smithi178.stderr:+ break 2024-01-11T21:47:41.587 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:47:41.627 INFO:tasks.workunit.client.0.smithi178.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2024-01-11T21:47:41.632 INFO:tasks.workunit.client.0.smithi178.stdout:NAME PROVISIONED USED 2024-01-11T21:47:41.632 INFO:tasks.workunit.client.0.smithi178.stdout:test1 64 MiB 64 MiB 2024-01-11T21:47:41.636 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 '!=' 0 ']' 2024-01-11T21:47:41.636 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:47:41.723 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete... Removing image: 100% complete...done. 2024-01-11T21:47:41.726 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:47:41.726 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:47:41.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:47:41.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:47:41.761 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:47:41.761 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --thick-provision -s 4G test1 2024-01-11T21:47:41.800 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:41.794+0000 7f91ad3234c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:47:42.572 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 1% complete... Thick provisioning: 2% complete... Thick provisioning: 3% complete... Thick provisioning: 4% complete... Thick provisioning: 5% complete... Thick provisioning: 6% complete... Thick provisioning: 7% complete... Thick provisioning: 8% complete... Thick provisioning: 9% complete... Thick provisioning: 10% complete...2024-01-11T21:47:42.566+0000 7f91abe59700 0 -- 172.21.15.178:0/2084187924 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5624bedeed50 msgr2=0x5624bee8f1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:42.679 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 11% complete... Thick provisioning: 12% complete...2024-01-11T21:47:42.674+0000 7f91ab658700 0 -- 172.21.15.178:0/2084187924 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5624beead0d0 msgr2=0x5624beeaf4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:47.941 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 13% complete... Thick provisioning: 14% complete... Thick provisioning: 15% complete... Thick provisioning: 16% complete... Thick provisioning: 17% complete... Thick provisioning: 18% complete... Thick provisioning: 19% complete... Thick provisioning: 20% complete... Thick provisioning: 21% complete... Thick provisioning: 22% complete... Thick provisioning: 23% complete... Thick provisioning: 24% complete... Thick provisioning: 25% complete... Thick provisioning: 26% complete... Thick provisioning: 27% complete... Thick provisioning: 28% complete... Thick provisioning: 29% complete... Thick provisioning: 30% complete... Thick provisioning: 31% complete... Thick provisioning: 32% complete... Thick provisioning: 33% complete... Thick provisioning: 34% complete... Thick provisioning: 35% complete... Thick provisioning: 36% complete... Thick provisioning: 37% complete... Thick provisioning: 38% complete... Thick provisioning: 39% complete... Thick provisioning: 40% complete... Thick provisioning: 41% complete... Thick provisioning: 42% complete... Thick provisioning: 43% complete... Thick provisioning: 44% complete... Thick provisioning: 45% complete... Thick provisioning: 46% complete... Thick provisioning: 47% complete... Thick provisioning: 48% complete... Thick provisioning: 49% complete... Thick provisioning: 50% complete... Thick provisioning: 51% complete... Thick provisioning: 52% complete... Thick provisioning: 53% complete... Thick provisioning: 54% complete... Thick provisioning: 55% complete... Thick provisioning: 56% complete... Thick provisioning: 57% complete... Thick provisioning: 58% complete... Thick provisioning: 59% complete... Thick provisioning: 60% complete... Thick provisioning: 61% complete... Thick provisioning: 62% complete... Thick provisioning: 63% complete... Thick provisioning: 64% complete... Thick provisioning: 65% complete... Thick provisioning: 66% complete... Thick provisioning: 67% complete... Thick provisioning: 68% complete... Thick provisioning: 69% complete... Thick provisioning: 70% complete... Thick provisioning: 71% complete... Thick provisioning: 72% complete... Thick provisioning: 73% complete... Thick provisioning: 74% complete... Thick provisioning: 75% complete... Thick provisioning: 76% complete... Thick provisioning: 77% complete... Thick provisioning: 78% complete... Thick provisioning: 79% complete... Thick provisioning: 80% complete... Thick provisioning: 81% complete... Thick provisioning: 82% complete... Thick provisioning: 83% complete... Thick provisioning: 84% complete... Thick provisioning: 85% complete... Thick provisioning: 86% complete... Thick provisioning: 87% complete... Thick provisioning: 88% complete... Thick provisioning: 89% complete... Thick provisioning: 90% complete... Thick provisioning: 91% complete... Thick provisioning: 92% complete... Thick provisioning: 93% complete... Thick provisioning: 94% complete... Thick provisioning: 95% complete... Thick provisioning: 96% complete... Thick provisioning: 97% complete... Thick provisioning: 98% complete... Thick provisioning: 99% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2024-01-11T21:47:47.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ count=0 2024-01-11T21:47:47.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret= 2024-01-11T21:47:47.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 -lt 10 ']' 2024-01-11T21:47:47.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:47:47.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:47:47.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ tr -s ' ' 2024-01-11T21:47:47.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ cut -d ' ' -f 4-5 2024-01-11T21:47:47.950 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^4 GiB' 2024-01-11T21:47:47.987 INFO:tasks.workunit.client.0.smithi178.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2024-01-11T21:47:48.004 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:48.002+0000 7fdab208b700 0 -- 172.21.15.178:0/1267342192 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5607263f4bc0 msgr2=0x560726404060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:48.008 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:48.002+0000 7fdab188a700 0 -- 172.21.15.178:0/1267342192 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x560726407720 msgr2=0x560726409b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:48.146 INFO:tasks.workunit.client.0.smithi178.stdout:4 GiB 2024-01-11T21:47:48.147 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret=0 2024-01-11T21:47:48.147 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 = 0 ']' 2024-01-11T21:47:48.147 INFO:tasks.workunit.client.0.smithi178.stderr:+ break 2024-01-11T21:47:48.147 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:47:48.181 INFO:tasks.workunit.client.0.smithi178.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2024-01-11T21:47:48.198 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:48.194+0000 7f6f1a630700 0 -- 172.21.15.178:0/3997634890 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x556c263e28e0 msgr2=0x556c263e4cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:48.202 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:47:48.198+0000 7f6f1ae31700 0 -- 172.21.15.178:0/3997634890 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x556c26334ca0 msgr2=0x556c263c3c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:48.336 INFO:tasks.workunit.client.0.smithi178.stdout:NAME PROVISIONED USED 2024-01-11T21:47:48.336 INFO:tasks.workunit.client.0.smithi178.stdout:test1 4 GiB 4 GiB 2024-01-11T21:47:48.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 '!=' 0 ']' 2024-01-11T21:47:48.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:47:48.485 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete...2024-01-11T21:47:48.482+0000 7f70c677c700 0 -- 172.21.15.178:0/2918718069 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5607c0d777d0 msgr2=0x5607c0d79bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:48.506 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 11% complete... Removing image: 12% complete...2024-01-11T21:47:48.502+0000 7f70c6f7d700 0 -- 172.21.15.178:0/2918718069 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5607c0cbb390 msgr2=0x5607c0d76d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:49.318 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-01-11T21:47:49.321 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:47:49.321 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:47:49.321 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:47:49.321 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:47:49.360 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:47:49.360 INFO:tasks.workunit.client.0.smithi178.stderr:+ RBD_CREATE_ARGS='--image-format 2' 2024-01-11T21:47:49.360 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_others 2024-01-11T21:47:49.360 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing import, export, resize, and snapshots...' 2024-01-11T21:47:49.361 INFO:tasks.workunit.client.0.smithi178.stdout:testing import, export, resize, and snapshots... 2024-01-11T21:47:49.361 INFO:tasks.workunit.client.0.smithi178.stderr:+ TMP_FILES='/tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1' 2024-01-11T21:47:49.361 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:47:49.361 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:49.481 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:49.608 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:49.726 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:49.848 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:49.981 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:50.109 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:50.243 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:50.372 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:50.503 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:50.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:50.765 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:50.901 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:51.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:51.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:51.306 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:51.438 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:51.572 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:47:51.711 INFO:tasks.workunit.client.0.smithi178.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-01-11T21:47:51.712 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/sh of=/tmp/img1 bs=1k count=1 seek=10 2024-01-11T21:47:51.713 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-01-11T21:47:51.713 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-01-11T21:47:51.713 INFO:tasks.workunit.client.0.smithi178.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000159134 s, 6.4 MB/s 2024-01-11T21:47:51.713 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/dd of=/tmp/img1 bs=1k count=10 seek=100 2024-01-11T21:47:51.715 INFO:tasks.workunit.client.0.smithi178.stderr:10+0 records in 2024-01-11T21:47:51.715 INFO:tasks.workunit.client.0.smithi178.stderr:10+0 records out 2024-01-11T21:47:51.715 INFO:tasks.workunit.client.0.smithi178.stderr:10240 bytes (10 kB, 10 KiB) copied, 0.00022199 s, 46.1 MB/s 2024-01-11T21:47:51.715 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/rm of=/tmp/img1 bs=1k count=100 seek=1000 2024-01-11T21:47:51.717 INFO:tasks.workunit.client.0.smithi178.stderr:70+1 records in 2024-01-11T21:47:51.717 INFO:tasks.workunit.client.0.smithi178.stderr:70+1 records out 2024-01-11T21:47:51.717 INFO:tasks.workunit.client.0.smithi178.stderr:72056 bytes (72 kB, 70 KiB) copied, 0.000578448 s, 125 MB/s 2024-01-11T21:47:51.717 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/ls of=/tmp/img1 bs=1k seek=10000 2024-01-11T21:47:51.719 INFO:tasks.workunit.client.0.smithi178.stderr:138+1 records in 2024-01-11T21:47:51.719 INFO:tasks.workunit.client.0.smithi178.stderr:138+1 records out 2024-01-11T21:47:51.719 INFO:tasks.workunit.client.0.smithi178.stderr:142144 bytes (142 kB, 139 KiB) copied, 0.000871371 s, 163 MB/s 2024-01-11T21:47:51.719 INFO:tasks.workunit.client.0.smithi178.stderr:+ dd if=/bin/ln of=/tmp/img1 bs=1k seek=100000 2024-01-11T21:47:51.721 INFO:tasks.workunit.client.0.smithi178.stderr:74+1 records in 2024-01-11T21:47:51.721 INFO:tasks.workunit.client.0.smithi178.stderr:74+1 records out 2024-01-11T21:47:51.721 INFO:tasks.workunit.client.0.smithi178.stderr:76160 bytes (76 kB, 74 KiB) copied, 0.000614187 s, 124 MB/s 2024-01-11T21:47:51.721 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import --image-format 2 /tmp/img1 testimg1 2024-01-11T21:47:51.900 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image: 4% complete... Importing image: 8% complete... Importing image: 12% complete... Importing image: 16% complete... Importing image: 20% complete... Importing image: 24% complete... Importing image: 28% complete... Importing image: 32% complete... Importing image: 36% complete... Importing image: 40% complete... Importing image: 45% complete... Importing image: 49% complete... Importing image: 53% complete... Importing image: 57% complete... Importing image: 61% complete... Importing image: 65% complete... Importing image: 69% complete... Importing image: 73% complete... Importing image: 77% complete... Importing image: 81% complete... Importing image: 85% complete... Importing image: 90% complete... Importing image: 94% complete... Importing image: 98% complete... Importing image: 100% complete...done. 2024-01-11T21:47:51.904 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd resize testimg1 --size=256 --allow-shrink 2024-01-11T21:47:51.979 INFO:tasks.workunit.client.0.smithi178.stderr: Resizing image: 100% complete...done. 2024-01-11T21:47:51.996 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img2 2024-01-11T21:47:52.138 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:52.142 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1 --snap=snap1 2024-01-11T21:47:52.574 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:47:52.588 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd resize testimg1 --size=128 2024-01-11T21:47:52.643 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: shrinking an image is only allowed with the --allow-shrink flag 2024-01-11T21:47:52.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:47:52.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd resize testimg1 --size=128 --allow-shrink 2024-01-11T21:47:52.730 INFO:tasks.workunit.client.0.smithi178.stderr: Resizing image: 50% complete... Resizing image: 51% complete... Resizing image: 53% complete... Resizing image: 54% complete... Resizing image: 56% complete... Resizing image: 57% complete... Resizing image: 59% complete... Resizing image: 60% complete... Resizing image: 62% complete... Resizing image: 64% complete... Resizing image: 65% complete... Resizing image: 67% complete... Resizing image: 68% complete... Resizing image: 70% complete... Resizing image: 71% complete... Resizing image: 73% complete... Resizing image: 75% complete... Resizing image: 76% complete... Resizing image: 78% complete... Resizing image: 79% complete... Resizing image: 81% complete... Resizing image: 82% complete... Resizing image: 84% complete... Resizing image: 85% complete... Resizing image: 87% complete... Resizing image: 89% complete... Resizing image: 90% complete... Resizing image: 92% complete... Resizing image: 93% complete... Resizing image: 95% complete... Resizing image: 96% complete... Resizing image: 98% complete... Resizing image: 100% complete...done. 2024-01-11T21:47:52.749 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img3 2024-01-11T21:47:52.849 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:52.853 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg1 2024-01-11T21:47:52.853 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:52.901 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:52.901 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg1 2024-01-11T21:47:52.902 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:52.963 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:52.963 INFO:tasks.workunit.client.0.smithi178.stderr:+ rm -rf /tmp/diff-testimg1-1 /tmp/diff-testimg1-2 2024-01-11T21:47:52.965 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export-diff testimg1 --snap=snap1 /tmp/diff-testimg1-1 2024-01-11T21:47:53.079 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 37% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:53.083 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export-diff testimg1 --from-snap=snap1 /tmp/diff-testimg1-2 2024-01-11T21:47:53.157 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 100% complete...done. 2024-01-11T21:47:53.162 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size=1 testimg-diff1 2024-01-11T21:47:53.218 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-1 testimg-diff1 2024-01-11T21:47:53.591 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image diff: 22% complete... Importing image diff: 63% complete... Importing image diff: 99% complete... Importing image diff: 100% complete...done. 2024-01-11T21:47:53.606 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-2 testimg-diff1 2024-01-11T21:47:53.683 INFO:tasks.workunit.client.0.smithi178.stderr: Importing image diff: 68% complete... Importing image diff: 96% complete... Importing image diff: 100% complete...done. 2024-01-11T21:47:53.701 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg1 2024-01-11T21:47:53.702 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:53.764 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:53.765 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg1 2024-01-11T21:47:53.765 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:53.827 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:53.828 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:53.828 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff1 2024-01-11T21:47:53.884 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:53.885 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-01-11T21:47:53.885 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:53.948 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:53.948 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg1 --snap=snap1 testimg2 2024-01-11T21:47:54.111 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 3% complete... Image copy: 37% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:54.115 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg1 testimg3 2024-01-11T21:47:54.264 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:54.267 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg-diff1 --sparse-size 768K --snap=snap1 testimg-diff2 2024-01-11T21:47:54.409 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 3% complete... Image copy: 37% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:54.413 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd copy testimg-diff1 --sparse-size 768K testimg-diff3 2024-01-11T21:47:54.563 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-01-11T21:47:54.568 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg2 2024-01-11T21:47:54.568 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:54.612 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg3 2024-01-11T21:47:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:54.668 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:54.669 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff2 2024-01-11T21:47:54.669 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:54.724 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:54.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff3 2024-01-11T21:47:54.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:54.783 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:54.783 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep copy testimg1 testimg4 2024-01-11T21:47:55.652 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-01-11T21:47:55.657 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep copy testimg1 --snap=snap1 testimg5 2024-01-11T21:47:56.661 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-01-11T21:47:56.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg4 2024-01-11T21:47:56.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 128 MiB' 2024-01-11T21:47:56.726 INFO:tasks.workunit.client.0.smithi178.stdout: size 128 MiB in 32 objects 2024-01-11T21:47:56.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg5 2024-01-11T21:47:56.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:56.777 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:56.778 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg4 2024-01-11T21:47:56.778 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:47:56.779 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:47:56.779 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:47:56.840 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:47:56.841 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg4 2024-01-11T21:47:56.841 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '.*snap1.*' 2024-01-11T21:47:56.894 INFO:tasks.workunit.client.0.smithi178.stdout: 26 snap1 256 MiB Thu Jan 11 21:47:55 2024 2024-01-11T21:47:56.895 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img1.new 2024-01-11T21:47:57.007 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:57.011 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg2 /tmp/img2.new 2024-01-11T21:47:57.153 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:57.158 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg3 /tmp/img3.new 2024-01-11T21:47:57.259 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:57.261 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.new 2024-01-11T21:47:57.366 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:57.370 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff2 /tmp/img-diff2.new 2024-01-11T21:47:57.494 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:57.498 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff3 /tmp/img-diff3.new 2024-01-11T21:47:57.812 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:57.816 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img2.new 2024-01-11T21:47:58.048 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img3 /tmp/img3.new 2024-01-11T21:47:58.165 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img-diff2.new 2024-01-11T21:47:58.364 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img3 /tmp/img-diff3.new 2024-01-11T21:47:58.465 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rollback --snap=snap1 testimg1 2024-01-11T21:47:58.619 INFO:tasks.workunit.client.0.smithi178.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-01-11T21:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rollback --snap=snap1 testimg-diff1 2024-01-11T21:47:58.783 INFO:tasks.workunit.client.0.smithi178.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-01-11T21:47:58.795 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:58.795 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg1 2024-01-11T21:47:58.854 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:58.855 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg-diff1 2024-01-11T21:47:58.855 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:47:58.909 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:47:58.909 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg1 /tmp/img1.snap1 2024-01-11T21:47:59.034 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:59.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.snap1 2024-01-11T21:47:59.168 INFO:tasks.workunit.client.0.smithi178.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-01-11T21:47:59.172 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img1.snap1 2024-01-11T21:47:59.362 INFO:tasks.workunit.client.0.smithi178.stderr:+ cmp /tmp/img2 /tmp/img-diff1.snap1 2024-01-11T21:47:59.555 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm testimg2 2024-01-11T21:47:59.736 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-01-11T21:47:59.734+0000 7f77ad9a0700 0 -- 172.21.15.178:0/627200450 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f778c008740 msgr2=0x7f778c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:59.761 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:59.766 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm testimg3 2024-01-11T21:47:59.967 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-01-11T21:47:59.962+0000 7f210c5e9700 0 -- 172.21.15.178:0/2286057091 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55875350c020 msgr2=0x5587535acbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:47:59.981 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:47:59.986 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create testimg2 -s 0 2024-01-11T21:48:00.028 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:00.022+0000 7f6c220aa4c0 -1 librbd: Forced V1 image creation. 2024-01-11T21:48:00.040 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd cp testimg2 testimg3 2024-01-11T21:48:00.122 INFO:tasks.workunit.client.0.smithi178.stderr: Image copy: 100% complete...done. 2024-01-11T21:48:00.127 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep cp testimg2 testimg6 2024-01-11T21:48:00.196 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 100% complete...done. 2024-01-11T21:48:00.200 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm --snap=snap1 testimg1 2024-01-11T21:48:00.551 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:00.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm --snap=snap1 testimg-diff1 2024-01-11T21:48:01.567 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:01.581 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg1 2024-01-11T21:48:01.581 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-01-11T21:48:01.636 INFO:tasks.workunit.client.0.smithi178.stdout:error setting snapshot context: (2) No such file or directory 2024-01-11T21:48:01.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-01-11T21:48:01.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-01-11T21:48:01.689 INFO:tasks.workunit.client.0.smithi178.stdout:error setting snapshot context: (2) No such file or directory 2024-01-11T21:48:01.689 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd sparsify testimg1 2024-01-11T21:48:01.788 INFO:tasks.workunit.client.0.smithi178.stderr: Image sparsify: 1% complete... Image sparsify: 3% complete... Image sparsify: 4% complete... Image sparsify: 6% complete... Image sparsify: 7% complete... Image sparsify: 9% complete... Image sparsify: 10% complete... Image sparsify: 12% complete... Image sparsify: 14% complete... Image sparsify: 15% complete... Image sparsify: 17% complete... Image sparsify: 18% complete... Image sparsify: 20% complete... Image sparsify: 21% complete... Image sparsify: 23% complete... Image sparsify: 25% complete... Image sparsify: 26% complete... Image sparsify: 28% complete... Image sparsify: 29% complete... Image sparsify: 31% complete... Image sparsify: 32% complete... Image sparsify: 34% complete... Image sparsify: 35% complete... Image sparsify: 37% complete... Image sparsify: 39% complete... Image sparsify: 40% complete... Image sparsify: 42% complete... Image sparsify: 43% complete... Image sparsify: 45% complete... Image sparsify: 46% complete... Image sparsify: 48% complete... Image sparsify: 50% complete... Image sparsify: 51% complete... Image sparsify: 53% complete... Image sparsify: 54% complete... Image sparsify: 56% complete... Image sparsify: 57% complete... Image sparsify: 59% complete... Image sparsify: 60% complete... Image sparsify: 62% complete... Image sparsify: 64% complete... Image sparsify: 65% complete... Image sparsify: 67% complete... Image sparsify: 68% complete... Image sparsify: 70% complete... Image sparsify: 71% complete... Image sparsify: 73% complete... Image sparsify: 75% complete... Image sparsify: 76% complete... Image sparsify: 78% complete... Image sparsify: 79% complete... Image sparsify: 81% complete... Image sparsify: 82% complete... Image sparsify: 84% complete... Image sparsify: 85% complete... Image sparsify: 87% complete... Image sparsify: 89% complete... Image sparsify: 90% complete... Image sparsify: 92% complete... Image sparsify: 93% complete... Image sparsify: 95% complete... Image sparsify: 96% complete... Image sparsify: 98% complete... Image sparsify: 100% complete...done. 2024-01-11T21:48:01.803 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:48:01.803 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:02.105 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:02.239 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:02.466 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:02.794 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:04.031 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:04.252 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:04.543 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:04.841 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:05.123 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:05.257 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:05.390 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:05.521 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:05.655 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:05.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:05.928 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:06.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:06.210 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:06.333 INFO:tasks.workunit.client.0.smithi178.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-01-11T21:48:06.435 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_locking 2024-01-11T21:48:06.436 INFO:tasks.workunit.client.0.smithi178.stdout:testing locking... 2024-01-11T21:48:06.436 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing locking...' 2024-01-11T21:48:06.436 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:48:06.436 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:06.567 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:06.706 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:06.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:06.970 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:07.102 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:07.242 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:07.383 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:07.515 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:07.645 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:07.783 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:07.921 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:08.054 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:08.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:08.534 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:08.668 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:08.809 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:08.947 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:09.086 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test1 2024-01-11T21:48:09.171 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:09.172 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:09.172 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:48:09.224 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:09.224 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id 2024-01-11T21:48:09.296 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:09.296 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 1 ' 2024-01-11T21:48:09.352 INFO:tasks.workunit.client.0.smithi178.stdout:There is 1 exclusive lock on this image. 2024-01-11T21:48:09.353 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd lock list test1 2024-01-11T21:48:09.353 INFO:tasks.workunit.client.0.smithi178.stderr:++ tail -n 1 2024-01-11T21:48:09.353 INFO:tasks.workunit.client.0.smithi178.stderr:++ awk '{print $1;}' 2024-01-11T21:48:09.410 INFO:tasks.workunit.client.0.smithi178.stderr:+ LOCKER=client.8449 2024-01-11T21:48:09.411 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock remove test1 id client.8449 2024-01-11T21:48:09.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:09.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:09.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:48:09.621 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:09.622 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id --shared tag 2024-01-11T21:48:09.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:09.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 1 ' 2024-01-11T21:48:09.755 INFO:tasks.workunit.client.0.smithi178.stdout:There is 1 shared lock on this image. 2024-01-11T21:48:09.756 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id --shared tag 2024-01-11T21:48:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 2 ' 2024-01-11T21:48:09.885 INFO:tasks.workunit.client.0.smithi178.stdout:There are 2 shared locks on this image. 2024-01-11T21:48:09.885 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock add test1 id2 --shared tag 2024-01-11T21:48:09.955 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:09.955 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep ' 3 ' 2024-01-11T21:48:10.016 INFO:tasks.workunit.client.0.smithi178.stdout:There are 3 shared locks on this image. 2024-01-11T21:48:10.016 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:10.016 INFO:tasks.workunit.client.0.smithi178.stderr:+ tail -n 1 2024-01-11T21:48:10.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ awk '{print $2, $1;}' 2024-01-11T21:48:10.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ xargs rbd lock remove test1 2024-01-11T21:48:10.759 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info test1 2024-01-11T21:48:10.760 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -qE 'features:.*exclusive' 2024-01-11T21:48:10.820 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd lock list test1 2024-01-11T21:48:10.877 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' -n 'There are 2 shared locks on this image. 2024-01-11T21:48:10.877 INFO:tasks.workunit.client.0.smithi178.stderr:Lock tag: tag 2024-01-11T21:48:10.877 INFO:tasks.workunit.client.0.smithi178.stderr:Locker ID Address 2024-01-11T21:48:10.877 INFO:tasks.workunit.client.0.smithi178.stderr:client.8464 id 172.21.15.178:0/49490945 2024-01-11T21:48:10.877 INFO:tasks.workunit.client.0.smithi178.stderr:client.8470 id 172.21.15.178:0/1229864443' ']' 2024-01-11T21:48:10.878 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:10.878 INFO:tasks.workunit.client.0.smithi178.stderr:+ tail -n 1 2024-01-11T21:48:10.878 INFO:tasks.workunit.client.0.smithi178.stderr:+ awk '{print $2, $1;}' 2024-01-11T21:48:10.878 INFO:tasks.workunit.client.0.smithi178.stderr:+ xargs rbd lock remove test1 2024-01-11T21:48:11.769 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd lock list test1 2024-01-11T21:48:11.826 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' -n 'There is 1 shared lock on this image. 2024-01-11T21:48:11.826 INFO:tasks.workunit.client.0.smithi178.stderr:Lock tag: tag 2024-01-11T21:48:11.826 INFO:tasks.workunit.client.0.smithi178.stderr:Locker ID Address 2024-01-11T21:48:11.826 INFO:tasks.workunit.client.0.smithi178.stderr:client.8464 id 172.21.15.178:0/49490945' ']' 2024-01-11T21:48:11.827 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd lock list test1 2024-01-11T21:48:11.827 INFO:tasks.workunit.client.0.smithi178.stderr:+ tail -n 1 2024-01-11T21:48:11.827 INFO:tasks.workunit.client.0.smithi178.stderr:+ awk '{print $2, $1;}' 2024-01-11T21:48:11.827 INFO:tasks.workunit.client.0.smithi178.stderr:+ xargs rbd lock remove test1 2024-01-11T21:48:12.776 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd lock list test1 2024-01-11T21:48:12.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' -n '' ']' 2024-01-11T21:48:12.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:48:13.039 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:13.034+0000 7f2f26a0c700 0 -- 172.21.15.178:0/1094942472 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f2f0c008740 msgr2=0x7f2f0c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:48:13.040 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:13.045 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_clone 2024-01-11T21:48:13.045 INFO:tasks.workunit.client.0.smithi178.stdout:testing clone... 2024-01-11T21:48:13.046 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing clone...' 2024-01-11T21:48:13.046 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:48:13.046 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:13.180 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:13.315 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:13.451 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:13.587 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:13.720 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:13.854 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:13.989 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:14.129 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:14.257 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:14.393 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:14.529 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:14.669 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:14.804 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:14.928 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:15.061 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:15.197 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:15.331 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:15.463 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create test1 --image-format 2 -s 1 2024-01-11T21:48:15.533 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@s1 2024-01-11T21:48:16.575 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:16.589 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect test1@s1 2024-01-11T21:48:16.655 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool create rbd2 8 2024-01-11T21:48:17.668 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd2' already exists 2024-01-11T21:48:17.676 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd pool init rbd2 2024-01-11T21:48:20.581 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone test1@s1 rbd2/clone 2024-01-11T21:48:20.682 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd -p rbd2 ls 2024-01-11T21:48:20.682 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep clone 2024-01-11T21:48:20.727 INFO:tasks.workunit.client.0.smithi178.stdout:clone 2024-01-11T21:48:20.728 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd -p rbd2 ls -l 2024-01-11T21:48:20.728 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep clone 2024-01-11T21:48:20.728 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1@s1 2024-01-11T21:48:20.801 INFO:tasks.workunit.client.0.smithi178.stdout:clone 1 MiB rbd/test1@s1 2 2024-01-11T21:48:20.802 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:20.802 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v clone 2024-01-11T21:48:20.845 INFO:tasks.workunit.client.0.smithi178.stdout:test1 2024-01-11T21:48:20.845 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd flatten rbd2/clone 2024-01-11T21:48:20.931 INFO:tasks.workunit.client.0.smithi178.stderr: Image flatten: 100% complete...done. 2024-01-11T21:48:20.944 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create rbd2/clone@s1 2024-01-11T21:48:21.597 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:21.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect rbd2/clone@s1 2024-01-11T21:48:21.670 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone rbd2/clone@s1 clone2 2024-01-11T21:48:21.791 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:21.791 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep clone2 2024-01-11T21:48:21.831 INFO:tasks.workunit.client.0.smithi178.stdout:clone2 2024-01-11T21:48:21.832 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:48:21.832 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep clone2 2024-01-11T21:48:21.832 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep rbd2/clone@s1 2024-01-11T21:48:21.909 INFO:tasks.workunit.client.0.smithi178.stdout:clone2 1 MiB rbd2/clone@s1 2 2024-01-11T21:48:21.910 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd -p rbd2 ls 2024-01-11T21:48:21.910 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v clone2 2024-01-11T21:48:21.954 INFO:tasks.workunit.client.0.smithi178.stdout:clone 2024-01-11T21:48:21.954 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm clone2 2024-01-11T21:48:22.154 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:22.149+0000 7fce3d432700 0 -- 172.21.15.178:0/2286347367 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5595cc57bf20 msgr2=0x5595cc61dbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:48:22.157 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:22.162 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect rbd2/clone@s1 2024-01-11T21:48:22.229 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm rbd2/clone@s1 2024-01-11T21:48:22.596 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:22.605 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm rbd2/clone 2024-01-11T21:48:22.759 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:22.753+0000 7fbe610ec700 0 -- 172.21.15.178:0/18198163 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fbe40008740 msgr2=0x7fbe4000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:48:22.766 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:22.772 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect test1@s1 2024-01-11T21:48:22.864 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm test1@s1 2024-01-11T21:48:23.609 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:23.627 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:48:23.851 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:23.845+0000 7f43f2f1a700 0 -- 172.21.15.178:0/2332741040 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f43d4006750 msgr2=0x7f43d4008bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:48:23.852 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:23.857 INFO:tasks.workunit.client.0.smithi178.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-01-11T21:48:24.715 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd2' does not exist 2024-01-11T21:48:24.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_trash 2024-01-11T21:48:24.724 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing trash...' 2024-01-11T21:48:24.724 INFO:tasks.workunit.client.0.smithi178.stdout:testing trash... 2024-01-11T21:48:24.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:48:24.725 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:24.852 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:24.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:25.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:25.389 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:25.520 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:25.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:25.777 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:25.914 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:26.051 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:26.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:26.320 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:26.457 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:26.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:26.729 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:26.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:27.006 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:27.145 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test1 2024-01-11T21:48:27.224 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test2 2024-01-11T21:48:27.297 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:27.298 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:48:27.330 INFO:tasks.workunit.client.0.smithi178.stdout:test1 2024-01-11T21:48:27.331 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:27.331 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test2 2024-01-11T21:48:27.370 INFO:tasks.workunit.client.0.smithi178.stdout:test2 2024-01-11T21:48:27.371 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:27.371 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:27.371 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:48:27.413 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:48:27.413 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:48:27.414 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test1.*2.*' 2024-01-11T21:48:27.473 INFO:tasks.workunit.client.0.smithi178.stdout:test1 1 MiB 2 2024-01-11T21:48:27.473 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:48:27.474 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test2.*2.*' 2024-01-11T21:48:27.534 INFO:tasks.workunit.client.0.smithi178.stdout:test2 1 MiB 2 2024-01-11T21:48:27.534 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv test1 2024-01-11T21:48:27.639 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:27.639 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test2 2024-01-11T21:48:27.678 INFO:tasks.workunit.client.0.smithi178.stdout:test2 2024-01-11T21:48:27.679 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:27.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:27.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:27.722 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:27.722 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:48:27.723 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test2.*2.*' 2024-01-11T21:48:27.775 INFO:tasks.workunit.client.0.smithi178.stdout:test2 1 MiB 2 2024-01-11T21:48:27.776 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:27.776 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:48:27.816 INFO:tasks.workunit.client.0.smithi178.stdout:2263ee689a44 test1 2024-01-11T21:48:27.817 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:27.817 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:27.818 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:27.855 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -l 2024-01-11T21:48:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test1.*USER.*' 2024-01-11T21:48:27.910 INFO:tasks.workunit.client.0.smithi178.stdout:2263ee689a44 test1 USER Thu Jan 11 21:48:27 2024 expired at Thu Jan 11 21:48:27 2024 2024-01-11T21:48:27.911 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -l 2024-01-11T21:48:27.911 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v 'protected until' 2024-01-11T21:48:27.971 INFO:tasks.workunit.client.0.smithi178.stdout:ID NAME SOURCE DELETED_AT STATUS PARENT 2024-01-11T21:48:27.971 INFO:tasks.workunit.client.0.smithi178.stdout:2263ee689a44 test1 USER Thu Jan 11 21:48:27 2024 expired at Thu Jan 11 21:48:27 2024 2024-01-11T21:48:27.972 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:48:27.972 INFO:tasks.workunit.client.0.smithi178.stderr:++ cut -d ' ' -f 1 2024-01-11T21:48:28.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=2263ee689a44 2024-01-11T21:48:28.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash rm 2263ee689a44 2024-01-11T21:48:28.130 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:28.136 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv test2 2024-01-11T21:48:28.239 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:48:28.239 INFO:tasks.workunit.client.0.smithi178.stderr:++ cut -d ' ' -f 1 2024-01-11T21:48:28.278 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=2266a9aa1388 2024-01-11T21:48:28.279 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info --image-id 2266a9aa1388 2024-01-11T21:48:28.279 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'rbd image '\''test2'\''' 2024-01-11T21:48:28.338 INFO:tasks.workunit.client.0.smithi178.stdout:rbd image 'test2': 2024-01-11T21:48:28.339 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash restore 2266a9aa1388 2024-01-11T21:48:28.403 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:28.403 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test2 2024-01-11T21:48:28.443 INFO:tasks.workunit.client.0.smithi178.stdout:test2 2024-01-11T21:48:28.443 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:48:28.444 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:28.444 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:28.486 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:28.487 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls -l 2024-01-11T21:48:28.487 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test2.*2.*' 2024-01-11T21:48:28.544 INFO:tasks.workunit.client.0.smithi178.stdout:test2 1 MiB 2 2024-01-11T21:48:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv test2 --expires-at '3600 sec' 2024-01-11T21:48:28.647 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: image test2 will expire at 2024-01-11T22:48:28.586665+0000 2024-01-11T21:48:28.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:28.652 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test2 2024-01-11T21:48:28.694 INFO:tasks.workunit.client.0.smithi178.stdout:2266a9aa1388 test2 2024-01-11T21:48:28.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:28.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:28.696 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:28.735 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:28.736 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test2.*USER.*protected until' 2024-01-11T21:48:28.736 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -l 2024-01-11T21:48:28.792 INFO:tasks.workunit.client.0.smithi178.stdout:2266a9aa1388 test2 USER Thu Jan 11 21:48:28 2024 protected until Thu Jan 11 22:48:28 2024 2024-01-11T21:48:28.793 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash rm 2266a9aa1388 2024-01-11T21:48:28.793 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'Deferment time has not expired' 2024-01-11T21:48:28.837 INFO:tasks.workunit.client.0.smithi178.stdout:Deferment time has not expired, please use --force if you 2024-01-11T21:48:28.825+0000 7fd6f61774c0 -1 librbd::api::Trash: remove: error: deferment time has not expired. 2024-01-11T21:48:28.837 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash rm --image-id 2266a9aa1388 --force 2024-01-11T21:48:28.950 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:28.958 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test1 2024-01-11T21:48:29.025 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@snap1 2024-01-11T21:48:29.572 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:29.586 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect test1@snap1 2024-01-11T21:48:29.655 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv test1 2024-01-11T21:48:29.766 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:29.766 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:48:29.807 INFO:tasks.workunit.client.0.smithi178.stdout:22bd6cf2e342 test1 2024-01-11T21:48:29.808 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:29.808 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:29.808 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:29.852 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:29.852 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -l 2024-01-11T21:48:29.853 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test1.*USER.*' 2024-01-11T21:48:29.911 INFO:tasks.workunit.client.0.smithi178.stdout:22bd6cf2e342 test1 USER Thu Jan 11 21:48:29 2024 expired at Thu Jan 11 21:48:29 2024 2024-01-11T21:48:29.911 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -l 2024-01-11T21:48:29.912 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v 'protected until' 2024-01-11T21:48:29.973 INFO:tasks.workunit.client.0.smithi178.stdout:ID NAME SOURCE DELETED_AT STATUS PARENT 2024-01-11T21:48:29.973 INFO:tasks.workunit.client.0.smithi178.stdout:22bd6cf2e342 test1 USER Thu Jan 11 21:48:29 2024 expired at Thu Jan 11 21:48:29 2024 2024-01-11T21:48:29.974 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:48:29.975 INFO:tasks.workunit.client.0.smithi178.stderr:++ cut -d ' ' -f 1 2024-01-11T21:48:30.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=22bd6cf2e342 2024-01-11T21:48:30.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls --image-id 22bd6cf2e342 2024-01-11T21:48:30.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:48:30.018 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:30.018 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:30.076 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:30.077 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls --image-id 22bd6cf2e342 2024-01-11T21:48:30.077 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '.*snap1.*' 2024-01-11T21:48:30.317 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:30.313+0000 7f231a8c1700 0 --2- 172.21.15.178:0/1765733497 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564612114e30 0x5646121b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:48:30.348 INFO:tasks.workunit.client.0.smithi178.stdout: 34 snap1 1 MiB yes Thu Jan 11 21:48:29 2024 2024-01-11T21:48:30.348 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect --image-id 22bd6cf2e342 --snap snap1 2024-01-11T21:48:30.433 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm --image-id 22bd6cf2e342 --snap snap1 2024-01-11T21:48:30.568 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:30.582 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls --image-id 22bd6cf2e342 2024-01-11T21:48:30.583 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:30.583 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:48:30.583 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:30.637 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:30.637 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash restore 22bd6cf2e342 2024-01-11T21:48:30.699 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@snap1 2024-01-11T21:48:31.583 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:31.599 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@snap2 2024-01-11T21:48:32.590 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:32.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls --image-id 22bd6cf2e342 2024-01-11T21:48:32.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:48:32.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:32.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:48:32.665 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:48:32.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge --image-id 22bd6cf2e342 2024-01-11T21:48:34.592 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:48:34.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls --image-id 22bd6cf2e342 2024-01-11T21:48:34.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:48:34.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:34.607 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:34.661 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:34.661 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm --rbd_move_to_trash_on_remove=true --rbd_move_to_trash_on_remove_expire_seconds=3600 test1 2024-01-11T21:48:34.759 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:34.765 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:34.765 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:48:34.804 INFO:tasks.workunit.client.0.smithi178.stdout:22bd6cf2e342 test1 2024-01-11T21:48:34.805 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:34.805 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:34.805 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:34.842 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:34.843 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -l 2024-01-11T21:48:34.843 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'test1.*USER.*protected until' 2024-01-11T21:48:34.903 INFO:tasks.workunit.client.0.smithi178.stdout:22bd6cf2e342 test1 USER Thu Jan 11 21:48:34 2024 protected until Thu Jan 11 22:48:34 2024 2024-01-11T21:48:34.904 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash rm 22bd6cf2e342 2024-01-11T21:48:34.904 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'Deferment time has not expired' 2024-01-11T21:48:34.948 INFO:tasks.workunit.client.0.smithi178.stdout:Deferment time has not expired, please use --force if you really want to remove the image 2024-01-11T21:48:34.948 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash rm --image-id 22bd6cf2e342 --force 2024-01-11T21:48:35.065 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:48:35.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:48:35.071 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:35.199 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:35.337 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:35.478 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:35.609 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:35.741 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:35.875 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.011 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.148 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.283 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.417 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.553 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.692 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.827 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:36.956 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:37.095 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:37.226 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:37.366 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:37.704 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_purge 2024-01-11T21:48:37.704 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing trash purge...' 2024-01-11T21:48:37.705 INFO:tasks.workunit.client.0.smithi178.stdout:testing trash purge... 2024-01-11T21:48:37.705 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:48:37.705 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:37.840 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:38.179 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:38.314 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:38.456 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:38.582 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:38.717 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:38.856 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:38.986 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:39.128 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:39.264 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:39.396 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:39.526 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:39.661 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:39.802 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:39.932 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:40.064 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:40.204 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:48:40.337 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:40.337 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:40.337 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:40.376 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:40.376 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:40.413 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete...done. 2024-01-11T21:48:40.417 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:48:40.706 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-01-11T21:48:40.771 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg1 2024-01-11T21:48:40.872 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg2 2024-01-11T21:48:40.978 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:40.978 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:40.978 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:48:41.020 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:48:41.023 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:41.210 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 50% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:48:41.215 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:41.215 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:41.215 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:41.251 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:41.251 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:48:41.324 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-01-11T21:48:41.557 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:41.553+0000 7fa20eb88700 0 --2- 172.21.15.178:0/2637249241 >> v2:172.21.15.178:3300/0 conn(0x56438463e490 0x56438463a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:48:41.601 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg1 --expires-at '1 hour' 2024-01-11T21:48:41.706 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: image testimg1 will expire at 2024-01-11T22:48:41.638656+0000 2024-01-11T21:48:41.711 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg2 --expires-at '3 hours' 2024-01-11T21:48:41.818 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: image testimg2 will expire at 2024-01-12T00:48:41.751820+0000 2024-01-11T21:48:41.825 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:41.825 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:41.825 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:48:41.866 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:48:41.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:41.903 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete...done. 2024-01-11T21:48:41.907 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:41.907 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:48:41.948 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:48:41.949 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge --expired-before 'now + 2 hours' 2024-01-11T21:48:42.076 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:48:42.081 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:42.081 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:42.082 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:42.123 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:42.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:48:42.124 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:42.166 INFO:tasks.workunit.client.0.smithi178.stdout:24012f216336 testimg2 2024-01-11T21:48:42.166 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge --expired-before 'now + 4 hours' 2024-01-11T21:48:42.287 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:48:42.293 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:42.293 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:42.293 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:42.336 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:42.336 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:48:42.400 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1@snap 2024-01-11T21:48:42.584 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:42.601 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-01-11T21:48:42.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-01-11T21:48:42.926 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg1 2024-01-11T21:48:43.023 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg2 2024-01-11T21:48:43.135 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg3 2024-01-11T21:48:43.251 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:43.294 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:43.294 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:48:43.295 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:48:43.295 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:43.295 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:48:43.581 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:48:43.581 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:43.582 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:43.582 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:43.619 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:43.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:43.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg1 2024-01-11T21:48:43.658 INFO:tasks.workunit.client.0.smithi178.stdout:242255373407 testimg1 2024-01-11T21:48:43.659 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:48:43.660 INFO:tasks.workunit.client.0.smithi178.stderr:++ awk '{ print $1 }' 2024-01-11T21:48:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=242255373407 2024-01-11T21:48:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge --image-id 242255373407 2024-01-11T21:48:44.582 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:48:44.593 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:44.713 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:48:44.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:44.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:44.718 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:44.757 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:44.758 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:48:44.831 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-01-11T21:48:44.903 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg2@snap 2024-01-11T21:48:45.603 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:45.620 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-01-11T21:48:45.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg1 2024-01-11T21:48:45.799 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg2 2024-01-11T21:48:45.905 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg3 2024-01-11T21:48:46.028 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:46.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:46.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:48:46.072 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:48:46.073 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:46.073 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:48:46.363 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:48:46.364 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:46.364 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:46.364 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:46.405 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:46.405 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:46.405 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:48:46.441 INFO:tasks.workunit.client.0.smithi178.stdout:245233d238bb testimg2 2024-01-11T21:48:46.443 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:48:46.443 INFO:tasks.workunit.client.0.smithi178.stderr:++ awk '{ print $1 }' 2024-01-11T21:48:46.484 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=245233d238bb 2024-01-11T21:48:46.484 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge --image-id 245233d238bb 2024-01-11T21:48:46.597 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:48:46.614 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:46.724 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:48:46.730 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:46.730 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:46.730 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:46.773 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:46.773 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:48:46.846 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-01-11T21:48:46.918 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-01-11T21:48:46.985 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg3@snap 2024-01-11T21:48:47.615 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:47.633 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg1 2024-01-11T21:48:47.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg2 2024-01-11T21:48:47.876 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg3 2024-01-11T21:48:47.979 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:47.979 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:48:47.979 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:48.017 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:48:48.019 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:48.019 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:48:48.313 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:48:48.314 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:48.314 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:48.314 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:48:48.353 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:48:48.353 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:48.354 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg3 2024-01-11T21:48:48.394 INFO:tasks.workunit.client.0.smithi178.stdout:24823e130a9e testimg3 2024-01-11T21:48:48.395 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:48:48.395 INFO:tasks.workunit.client.0.smithi178.stderr:++ awk '{ print $1 }' 2024-01-11T21:48:48.435 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=24823e130a9e 2024-01-11T21:48:48.435 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge --image-id 24823e130a9e 2024-01-11T21:48:48.609 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:48:48.624 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:48.736 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:48:48.740 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:48.740 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:48.741 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:48.778 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:48.778 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:48:48.849 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1@snap 2024-01-11T21:48:49.584 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:49.600 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-01-11T21:48:49.720 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg1@snap 2024-01-11T21:48:49.789 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:49.808 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-01-11T21:48:49.880 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg2@snap 2024-01-11T21:48:50.631 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:50.644 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-01-11T21:48:50.769 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-01-11T21:48:50.912 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg2@snap 2024-01-11T21:48:51.001 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:51.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg4@snap 2024-01-11T21:48:51.638 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:51.653 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-01-11T21:48:51.803 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg4@snap 2024-01-11T21:48:51.912 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:51.933 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg1 2024-01-11T21:48:52.045 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg2 2024-01-11T21:48:52.161 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg3 2024-01-11T21:48:52.270 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg4 2024-01-11T21:48:52.382 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:52.382 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:52.382 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:48:52.419 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:48:52.419 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:52.420 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:48:52.872 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:48:52.872 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:52.873 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:52.873 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:48:52.910 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:48:52.911 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:52.911 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg1 2024-01-11T21:48:52.951 INFO:tasks.workunit.client.0.smithi178.stdout:24a9f17d0ee5 testimg1 2024-01-11T21:48:52.952 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:48:52.952 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:52.993 INFO:tasks.workunit.client.0.smithi178.stdout:24af6ef5955d testimg2 2024-01-11T21:48:52.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:52.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg4 2024-01-11T21:48:53.032 INFO:tasks.workunit.client.0.smithi178.stdout:24bba979dea2 testimg4 2024-01-11T21:48:53.033 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg6 2024-01-11T21:48:53.138 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:53.138 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:53.138 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:48:53.176 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:48:53.177 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:53.177 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:48:53.993 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:48:53.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:53.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:53.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:48:54.037 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:48:54.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:54.038 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg1 2024-01-11T21:48:54.076 INFO:tasks.workunit.client.0.smithi178.stdout:24a9f17d0ee5 testimg1 2024-01-11T21:48:54.077 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:54.077 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:48:54.119 INFO:tasks.workunit.client.0.smithi178.stdout:24af6ef5955d testimg2 2024-01-11T21:48:54.120 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg5 2024-01-11T21:48:54.239 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:54.239 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:54.239 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:48:54.279 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:48:54.279 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:48:54.466 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:54.461+0000 7f8d9d653700 0 -- 172.21.15.178:0/2995719924 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f8d7c0547c0 msgr2=0x7f8d7c056ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:48:54.490 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:48:54.485+0000 7f8d9de54700 0 -- 172.21.15.178:0/2995719924 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x56311218d430 msgr2=0x56311222edb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:48:55.763 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 33% complete... Removing images: 66% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:48:55.768 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:48:55.768 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:48:55.768 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:48:55.807 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:48:55.807 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:48:55.879 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1@snap 2024-01-11T21:48:56.671 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:56.688 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-01-11T21:48:56.804 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg1@snap 2024-01-11T21:48:56.876 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:56.894 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-01-11T21:48:56.966 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg3@snap 2024-01-11T21:48:57.677 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:57.695 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg2@snap 2024-01-11T21:48:58.675 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:58.693 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-01-11T21:48:58.824 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-01-11T21:48:58.955 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg2@snap 2024-01-11T21:48:59.048 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:59.068 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg4@snap 2024-01-11T21:48:59.589 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:48:59.604 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-01-11T21:48:59.758 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg4@snap 2024-01-11T21:48:59.867 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:48:59.887 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg1 2024-01-11T21:49:00.001 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg2 2024-01-11T21:49:00.101 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg3 2024-01-11T21:49:00.199 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg4 2024-01-11T21:49:00.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:00.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:00.291 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:00.330 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:00.331 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:00.331 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:00.604 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:00.604 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:00.605 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:00.605 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:00.644 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:00.644 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg6 2024-01-11T21:49:00.761 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:00.761 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:00.761 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 5 2024-01-11T21:49:00.804 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-01-11T21:49:00.805 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:00.805 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:02.242 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:02.243 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:02.243 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:02.243 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:49:02.284 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:49:02.284 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:02.284 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg1 2024-01-11T21:49:02.322 INFO:tasks.workunit.client.0.smithi178.stdout:2509a7168cf testimg1 2024-01-11T21:49:02.323 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:02.323 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:49:02.362 INFO:tasks.workunit.client.0.smithi178.stdout:250f77e4e248 testimg2 2024-01-11T21:49:02.363 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:02.363 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg3 2024-01-11T21:49:02.404 INFO:tasks.workunit.client.0.smithi178.stdout:2515c8de84a8 testimg3 2024-01-11T21:49:02.404 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg5 2024-01-11T21:49:02.523 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:02.523 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:02.523 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:02.563 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:02.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:02.564 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:05.085 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:05.085 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:05.085 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:05.086 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:49:05.129 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:49:05.129 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:05.130 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg3 2024-01-11T21:49:05.172 INFO:tasks.workunit.client.0.smithi178.stdout:2515c8de84a8 testimg3 2024-01-11T21:49:05.173 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:49:05.173 INFO:tasks.workunit.client.0.smithi178.stderr:++ awk '{ print $1 }' 2024-01-11T21:49:05.218 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=2515c8de84a8 2024-01-11T21:49:05.218 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge --image-id 2515c8de84a8 2024-01-11T21:49:05.786 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:49:05.804 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:05.923 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:49:05.928 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:05.928 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:05.928 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:49:05.971 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:49:05.972 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:49:06.047 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1@snap 2024-01-11T21:49:06.801 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:06.817 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-01-11T21:49:06.943 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg1@snap 2024-01-11T21:49:07.016 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:07.035 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-01-11T21:49:07.108 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg2@snap 2024-01-11T21:49:07.814 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:07.832 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-01-11T21:49:07.966 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-01-11T21:49:08.105 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg2@snap 2024-01-11T21:49:08.195 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:08.216 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg4@snap 2024-01-11T21:49:08.823 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:08.842 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-01-11T21:49:08.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg4@snap 2024-01-11T21:49:09.097 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:09.116 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg1 2024-01-11T21:49:09.275 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:09.279 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg2 2024-01-11T21:49:09.428 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:09.433 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg3 2024-01-11T21:49:09.557 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg4 2024-01-11T21:49:09.711 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:09.716 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:09.716 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:09.716 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:09.753 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:09.754 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:09.754 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:10.236 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:10.237 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:10.237 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:10.237 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:49:10.275 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:49:10.275 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:10.276 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg1 2024-01-11T21:49:10.318 INFO:tasks.workunit.client.0.smithi178.stdout:2575894c8f2d testimg1 2024-01-11T21:49:10.319 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:10.319 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:49:10.359 INFO:tasks.workunit.client.0.smithi178.stdout:257bc0a54848 testimg2 2024-01-11T21:49:10.361 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg4 2024-01-11T21:49:10.361 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:10.403 INFO:tasks.workunit.client.0.smithi178.stdout:258784b25785 testimg4 2024-01-11T21:49:10.403 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg6 2024-01-11T21:49:10.516 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:10.516 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:10.516 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:10.557 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:10.558 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:10.558 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:12.036 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:12.037 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:12.037 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:12.037 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 2 2024-01-11T21:49:12.078 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-01-11T21:49:12.079 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:12.079 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg1 2024-01-11T21:49:12.120 INFO:tasks.workunit.client.0.smithi178.stdout:2575894c8f2d testimg1 2024-01-11T21:49:12.121 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:12.121 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:49:12.166 INFO:tasks.workunit.client.0.smithi178.stdout:257bc0a54848 testimg2 2024-01-11T21:49:12.167 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg5 2024-01-11T21:49:12.293 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:12.294 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:12.294 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:49:12.332 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:49:12.332 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:12.530 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:12.524+0000 7fd2aa372700 0 -- 172.21.15.178:0/2412553210 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x5632378369f0 msgr2=0x5632378e5f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:12.551 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:12.544+0000 7fd2a9b71700 0 -- 172.21.15.178:0/2412553210 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fd2840547c0 msgr2=0x7fd284056ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:13.975 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 33% complete... Removing images: 66% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:49:13.981 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:13.981 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:13.981 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:49:14.022 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:49:14.022 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-01-11T21:49:14.099 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1@snap 2024-01-11T21:49:14.859 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:14.873 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-01-11T21:49:14.986 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg1@snap 2024-01-11T21:49:15.055 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:15.072 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-01-11T21:49:15.145 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg3@snap 2024-01-11T21:49:15.864 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:15.881 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg2@snap 2024-01-11T21:49:16.871 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:16.887 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-01-11T21:49:17.017 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-01-11T21:49:17.157 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg2@snap 2024-01-11T21:49:17.247 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:17.266 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg4@snap 2024-01-11T21:49:17.880 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:17.899 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-01-11T21:49:18.049 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm testimg4@snap 2024-01-11T21:49:18.151 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:18.169 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg1 2024-01-11T21:49:18.403 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:18.400+0000 7ffb446bf700 0 --2- 172.21.15.178:0/3865699314 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55be82a6d760 0x55be82b0c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:49:18.531 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:18.535 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg2 2024-01-11T21:49:18.688 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:18.694 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg3 2024-01-11T21:49:18.804 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg4 2024-01-11T21:49:18.950 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:18.956 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:18.956 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:18.956 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:18.998 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:18.998 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:19.263 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:19.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:19.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:19.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:19.305 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:19.305 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg6 2024-01-11T21:49:19.426 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:19.427 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:19.427 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 5 2024-01-11T21:49:19.466 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-01-11T21:49:19.467 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:19.467 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:20.834 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:20.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:20.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:20.835 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 3 2024-01-11T21:49:20.877 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:49:20.878 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:20.878 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg1 2024-01-11T21:49:20.921 INFO:tasks.workunit.client.0.smithi178.stdout:25d5d4ff5a23 testimg1 2024-01-11T21:49:20.922 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:20.922 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg2 2024-01-11T21:49:20.967 INFO:tasks.workunit.client.0.smithi178.stdout:25da4bcd33d8 testimg2 2024-01-11T21:49:20.967 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:20.967 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg3 2024-01-11T21:49:21.008 INFO:tasks.workunit.client.0.smithi178.stdout:25e03c4fade4 testimg3 2024-01-11T21:49:21.008 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash mv testimg5 2024-01-11T21:49:21.120 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:21.120 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:21.120 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 4 2024-01-11T21:49:21.163 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-01-11T21:49:21.163 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:21.164 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'some expired images could not be removed' 2024-01-11T21:49:22.788 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: some expired images could not be removed 2024-01-11T21:49:22.789 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:22.789 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:22.789 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:49:22.828 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:49:22.829 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:22.829 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep testimg3 2024-01-11T21:49:22.870 INFO:tasks.workunit.client.0.smithi178.stdout:25e03c4fade4 testimg3 2024-01-11T21:49:22.871 INFO:tasks.workunit.client.0.smithi178.stderr:++ rbd trash ls 2024-01-11T21:49:22.872 INFO:tasks.workunit.client.0.smithi178.stderr:++ awk '{ print $1 }' 2024-01-11T21:49:22.914 INFO:tasks.workunit.client.0.smithi178.stderr:+ ID=25e03c4fade4 2024-01-11T21:49:22.914 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge --image-id 25e03c4fade4 2024-01-11T21:49:23.612 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:49:23.626 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash purge 2024-01-11T21:49:23.734 INFO:tasks.workunit.client.0.smithi178.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-01-11T21:49:23.739 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls 2024-01-11T21:49:23.739 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:23.739 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 0 2024-01-11T21:49:23.780 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:49:23.781 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_deep_copy_clone 2024-01-11T21:49:23.782 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing deep copy clone...' 2024-01-11T21:49:23.782 INFO:tasks.workunit.client.0.smithi178.stdout:testing deep copy clone... 2024-01-11T21:49:23.783 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:49:23.783 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:23.910 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:24.051 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:24.188 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:24.333 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:24.479 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:24.619 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:24.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:24.904 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:25.040 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:25.180 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:25.312 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:25.450 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:25.591 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:25.738 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:25.873 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:26.018 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:26.158 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:26.289 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create testimg1 --image-format 2 --size 256 2024-01-11T21:49:26.364 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg1 --snap=snap1 2024-01-11T21:49:26.635 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:26.654 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect testimg1@snap1 2024-01-11T21:49:26.727 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone testimg1@snap1 testimg2 2024-01-11T21:49:26.859 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg2@snap2 2024-01-11T21:49:27.636 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:27.654 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep copy testimg2 testimg3 2024-01-11T21:49:27.826 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:27.820+0000 7fc4c01bb700 0 -- 172.21.15.178:0/17695247 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fc4a40065c0 msgr2=0x7fc4a406ed20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:28.657 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete...2024-01-11T21:49:28.652+0000 7fc4c01bb700 0 -- 172.21.15.178:0/17695247 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fc4a0036f10 msgr2=0x7fc4a0039330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:28.658 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 100% complete...done. 2024-01-11T21:49:28.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg3 2024-01-11T21:49:28.665 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:49:28.735 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:49:28.736 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg3 2024-01-11T21:49:28.736 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'parent: rbd/testimg1@snap1' 2024-01-11T21:49:28.813 INFO:tasks.workunit.client.0.smithi178.stdout: parent: rbd/testimg1@snap1 2024-01-11T21:49:28.814 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg3 2024-01-11T21:49:28.814 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:49:28.814 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:28.815 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:49:28.887 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:49:28.888 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg3 2024-01-11T21:49:28.888 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '.*snap2.*' 2024-01-11T21:49:28.962 INFO:tasks.workunit.client.0.smithi178.stdout: 76 snap2 256 MiB Thu Jan 11 21:49:28 2024 2024-01-11T21:49:28.962 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg2 2024-01-11T21:49:28.962 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'features:.*deep-flatten' 2024-01-11T21:49:29.035 INFO:tasks.workunit.client.0.smithi178.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-01-11T21:49:29.036 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg3 2024-01-11T21:49:29.036 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'features:.*deep-flatten' 2024-01-11T21:49:29.104 INFO:tasks.workunit.client.0.smithi178.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-01-11T21:49:29.105 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd flatten testimg2 2024-01-11T21:49:29.202 INFO:tasks.workunit.client.0.smithi178.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2024-01-11T21:49:29.223 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd flatten testimg3 2024-01-11T21:49:29.313 INFO:tasks.workunit.client.0.smithi178.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2024-01-11T21:49:29.330 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect testimg1@snap1 2024-01-11T21:49:29.415 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge testimg2 2024-01-11T21:49:29.591 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:49:29.604 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap purge testimg3 2024-01-11T21:49:30.639 INFO:tasks.workunit.client.0.smithi178.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-01-11T21:49:30.655 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm testimg2 2024-01-11T21:49:30.882 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-01-11T21:49:30.876+0000 7f4f0f9d8700 0 -- 172.21.15.178:0/3366280830 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55d237fc4480 msgr2=0x55d237fa28e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:30.882 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:30.887 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm testimg3 2024-01-11T21:49:31.082 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-01-11T21:49:31.076+0000 7f65fcd00700 0 -- 172.21.15.178:0/824293487 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f65dc036f40 msgr2=0x7f65dc039360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:49:31.114 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:31.118 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect testimg1@snap1 2024-01-11T21:49:31.191 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone testimg1@snap1 testimg2 2024-01-11T21:49:31.314 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create testimg2@snap2 2024-01-11T21:49:31.656 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:31.671 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd deep copy --flatten testimg2 testimg3 2024-01-11T21:49:32.791 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 1% complete...2024-01-11T21:49:32.784+0000 7f138c4ec700 0 -- 172.21.15.178:0/688699498 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x562c6712fae0 msgr2=0x562c67196300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:32.860 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete...2024-01-11T21:49:32.852+0000 7f138c4ec700 0 -- 172.21.15.178:0/688699498 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f136c0376c0 msgr2=0x7f136c039aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:33.380 INFO:tasks.workunit.client.0.smithi178.stderr: Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-01-11T21:49:33.386 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg3 2024-01-11T21:49:33.386 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'size 256 MiB' 2024-01-11T21:49:33.439 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:49:33.439 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg3 2024-01-11T21:49:33.439 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v parent: 2024-01-11T21:49:33.497 INFO:tasks.workunit.client.0.smithi178.stdout:rbd image 'testimg3': 2024-01-11T21:49:33.497 INFO:tasks.workunit.client.0.smithi178.stdout: size 256 MiB in 64 objects 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: order 22 (4 MiB objects) 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: snapshot_count: 1 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: id: 26ee2797c341 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: block_name_prefix: rbd_data.26ee2797c341 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: format: 2 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: op_features: 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: flags: 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: create_timestamp: Thu Jan 11 21:49:31 2024 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: access_timestamp: Thu Jan 11 21:49:31 2024 2024-01-11T21:49:33.498 INFO:tasks.workunit.client.0.smithi178.stdout: modify_timestamp: Thu Jan 11 21:49:31 2024 2024-01-11T21:49:33.499 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg3 2024-01-11T21:49:33.499 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -v SNAPID 2024-01-11T21:49:33.499 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:33.499 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 1 2024-01-11T21:49:33.561 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-01-11T21:49:33.562 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap ls testimg3 2024-01-11T21:49:33.562 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '.*snap2.*' 2024-01-11T21:49:33.624 INFO:tasks.workunit.client.0.smithi178.stdout: 80 snap2 256 MiB Thu Jan 11 21:49:32 2024 2024-01-11T21:49:33.624 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd info testimg2 2024-01-11T21:49:33.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'features:.*deep-flatten' 2024-01-11T21:49:33.705 INFO:tasks.workunit.client.0.smithi178.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-01-11T21:49:33.705 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd flatten testimg2 2024-01-11T21:49:33.805 INFO:tasks.workunit.client.0.smithi178.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2024-01-11T21:49:33.823 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect testimg1@snap1 2024-01-11T21:49:33.904 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:49:33.904 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:34.912 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:35.903 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:36.986 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:37.128 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:37.277 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:37.414 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:37.552 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:37.692 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:37.833 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:37.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:38.118 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:38.263 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:38.392 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:38.535 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:38.681 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:38.968 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:39.113 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_clone_v2 2024-01-11T21:49:39.114 INFO:tasks.workunit.client.0.smithi178.stdout:testing clone v2... 2024-01-11T21:49:39.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing clone v2...' 2024-01-11T21:49:39.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:49:39.114 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:39.260 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:39.399 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:39.544 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:39.698 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:39.837 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:39.980 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:40.123 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:40.270 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:40.418 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:40.573 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:40.717 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:40.861 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:41.011 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:41.155 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:41.317 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:41.460 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:41.596 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:41.740 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test1 2024-01-11T21:49:41.823 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@1 2024-01-11T21:49:42.661 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:42.681 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=1 test1@1 test2 2024-01-11T21:49:42.737 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:42.736+0000 7f990cfb6700 -1 librbd::image::CloneRequest: 0x562f6cbd1410 validate_parent: parent snapshot must be protected 2024-01-11T21:49:42.737 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: clone error: (22) Invalid argument 2024-01-11T21:49:42.741 INFO:tasks.workunit.client.0.smithi178.stderr:+ true 2024-01-11T21:49:42.741 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 test1@1 test2 2024-01-11T21:49:42.868 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=2 test1@1 test3 2024-01-11T21:49:42.994 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap protect test1@1 2024-01-11T21:49:43.060 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone --rbd-default-clone-format=1 test1@1 test4 2024-01-11T21:49:43.198 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd children test1@1 2024-01-11T21:49:43.198 INFO:tasks.workunit.client.0.smithi178.stderr:+ sort 2024-01-11T21:49:43.199 INFO:tasks.workunit.client.0.smithi178.stderr:+ tr '\n' ' ' 2024-01-11T21:49:43.199 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -E 'test2.*test3.*test4' 2024-01-11T21:49:43.265 INFO:tasks.workunit.client.0.smithi178.stdout:rbd/test2 rbd/test3 rbd/test4 2024-01-11T21:49:43.266 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd children --descendants test1 2024-01-11T21:49:43.266 INFO:tasks.workunit.client.0.smithi178.stderr:+ sort 2024-01-11T21:49:43.267 INFO:tasks.workunit.client.0.smithi178.stderr:+ tr '\n' ' ' 2024-01-11T21:49:43.267 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -E 'test2.*test3.*test4' 2024-01-11T21:49:43.363 INFO:tasks.workunit.client.0.smithi178.stdout:rbd/test2 rbd/test3 rbd/test4 2024-01-11T21:49:43.363 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd remove test4 2024-01-11T21:49:43.600 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:43.596+0000 7ff7b508b700 0 -- 172.21.15.178:0/2132851823 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x558334a1f190 msgr2=0x558334a41460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:49:43.611 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:43.615 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap unprotect test1@1 2024-01-11T21:49:43.694 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap remove test1@1 2024-01-11T21:49:43.773 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:43.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap list --all test1 2024-01-11T21:49:43.792 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep -E 'trash \(1\) *$' 2024-01-11T21:49:43.849 INFO:tasks.workunit.client.0.smithi178.stdout: 84 83b29354-12e2-44ab-b5c8-42a98d06c621 1 MiB Thu Jan 11 21:49:42 2024 trash (1) 2024-01-11T21:49:43.849 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@2 2024-01-11T21:49:44.612 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:44.632 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:49:44.632 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'image has snapshots' 2024-01-11T21:49:44.738 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed. 2024-01-11T21:49:44.738 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm test1@2 2024-01-11T21:49:45.608 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:45.624 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:49:45.625 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'linked clones' 2024-01-11T21:49:45.734 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2024-01-11T21:49:45.734 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test3 2024-01-11T21:49:45.960 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:45.955+0000 7fb68271d700 0 -- 172.21.15.178:0/1499751145 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55a802df6230 msgr2=0x55a802dfc260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:45.985 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:45.989 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:49:45.990 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep 'linked clones' 2024-01-11T21:49:46.096 INFO:tasks.workunit.client.0.smithi178.stdout:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2024-01-11T21:49:46.097 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd flatten test2 2024-01-11T21:49:46.633 INFO:tasks.workunit.client.0.smithi178.stderr: Image flatten: 100% complete...done. 2024-01-11T21:49:46.649 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap list --all test1 2024-01-11T21:49:46.650 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:46.650 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:49:46.707 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:49:46.707 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:49:46.950 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:46.947+0000 7fee319ae700 0 -- 172.21.15.178:0/3587587449 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x557a9e199500 msgr2=0x557a9e14fa70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:49:46.952 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:46.957 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:49:47.179 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:47.175+0000 7f6f5b29f700 0 -- 172.21.15.178:0/3344663870 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f6f40008740 msgr2=0x7f6f4000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:47.181 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 -s 1 test1 2024-01-11T21:49:47.265 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@1 2024-01-11T21:49:47.638 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:47.660 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap create test1@2 2024-01-11T21:49:48.645 INFO:tasks.workunit.client.0.smithi178.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-01-11T21:49:48.662 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone test1@1 test2 --rbd-default-clone-format 2 2024-01-11T21:49:48.805 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd clone test1@2 test3 --rbd-default-clone-format 2 2024-01-11T21:49:48.943 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm test1@1 2024-01-11T21:49:49.025 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:49.046 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd snap rm test1@2 2024-01-11T21:49:49.325 INFO:tasks.workunit.client.0.smithi178.stderr: Removing snap: 100% complete...done. 2024-01-11T21:49:49.342 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd rm test1 2024-01-11T21:49:49.343 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:49:49.438 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:49.435+0000 7ff1500304c0 -1 librbd::api::Image: remove: image has snapshots - not removing 2024-01-11T21:49:49.438 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 0% complete...failed. 2024-01-11T21:49:49.450 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2024-01-11T21:49:49.454 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:49:49.454 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 --rbd-move-parent-to-trash-on-remove=true 2024-01-11T21:49:49.613 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:49.618 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -a 2024-01-11T21:49:49.618 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:49:49.660 INFO:tasks.workunit.client.0.smithi178.stdout:2820b1ca2329 test1 2024-01-11T21:49:49.660 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test2 2024-01-11T21:49:49.873 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:49.871+0000 7f908ca89700 0 -- 172.21.15.178:0/3901320409 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x56125bacd2d0 msgr2=0x56125bb8a0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:49.910 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:49.907+0000 7f908ca89700 0 -- 172.21.15.178:0/3901320409 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7f906c036e40 msgr2=0x7f906c039260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:50.701 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:50.706 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -a 2024-01-11T21:49:50.706 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:49:50.752 INFO:tasks.workunit.client.0.smithi178.stdout:2820b1ca2329 test1 2024-01-11T21:49:50.752 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test3 2024-01-11T21:49:50.999 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:50.995+0000 7ff5f7cd0700 0 -- 172.21.15.178:0/170617025 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x564073ff0020 msgr2=0x564074090d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:51.628 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:51.623+0000 7ff5f74cf700 0 -- 172.21.15.178:0/170617025 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564073feb650 msgr2=0x56407408a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:51.739 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:51.745 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd trash ls -a 2024-01-11T21:49:51.745 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail grep test1 2024-01-11T21:49:51.745 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:49:51.787 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:49:51.788 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_thick_provision 2024-01-11T21:49:51.788 INFO:tasks.workunit.client.0.smithi178.stdout:testing thick provision... 2024-01-11T21:49:51.788 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing thick provision...' 2024-01-11T21:49:51.788 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:49:51.788 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:51.938 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:52.081 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:52.218 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:52.353 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:52.701 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:52.840 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:53.134 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:53.481 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:53.628 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:53.774 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:53.915 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:54.059 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:54.205 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:54.345 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:54.491 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:54.635 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:49:54.775 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --thick-provision -s 64M test1 2024-01-11T21:49:55.095 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 6% complete... Thick provisioning: 12% complete... Thick provisioning: 18% complete... Thick provisioning: 25% complete... Thick provisioning: 31% complete... Thick provisioning: 37% complete... Thick provisioning: 43% complete... Thick provisioning: 50% complete... Thick provisioning: 56% complete... Thick provisioning: 62% complete... Thick provisioning: 68% complete... Thick provisioning: 75% complete... Thick provisioning: 81% complete... Thick provisioning: 87% complete... Thick provisioning: 93% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2024-01-11T21:49:55.111 INFO:tasks.workunit.client.0.smithi178.stderr:+ count=0 2024-01-11T21:49:55.111 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret= 2024-01-11T21:49:55.111 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 -lt 10 ']' 2024-01-11T21:49:55.112 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:49:55.112 INFO:tasks.workunit.client.0.smithi178.stderr:+ tr -s ' ' 2024-01-11T21:49:55.112 INFO:tasks.workunit.client.0.smithi178.stderr:+ cut -d ' ' -f 4-5 2024-01-11T21:49:55.112 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^64 MiB' 2024-01-11T21:49:55.112 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:49:55.166 INFO:tasks.workunit.client.0.smithi178.stdout:64 MiB 2024-01-11T21:49:55.167 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret=0 2024-01-11T21:49:55.167 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 = 0 ']' 2024-01-11T21:49:55.167 INFO:tasks.workunit.client.0.smithi178.stderr:+ break 2024-01-11T21:49:55.167 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:49:55.401 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:55.399+0000 7f18089e9700 0 --2- 172.21.15.178:0/860761444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f9d597c170 0x55f9d5a091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-11T21:49:55.423 INFO:tasks.workunit.client.0.smithi178.stdout:NAME PROVISIONED USED 2024-01-11T21:49:55.423 INFO:tasks.workunit.client.0.smithi178.stdout:test1 64 MiB 64 MiB 2024-01-11T21:49:55.427 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 '!=' 0 ']' 2024-01-11T21:49:55.427 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:49:55.666 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-01-11T21:49:55.663+0000 7f42c71e1700 0 -- 172.21.15.178:0/1184332890 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x55cc3d42b4a0 msgr2=0x55cc3d4e7b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:55.715 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:49:55.711+0000 7f42c69e0700 0 -- 172.21.15.178:0/1184332890 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x55cc3d5580b0 msgr2=0x55cc3d57a3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:55.715 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 100% complete...done. 2024-01-11T21:49:55.720 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:49:55.720 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:49:55.720 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:49:55.720 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:49:55.762 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:49:55.762 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --thick-provision -s 4G test1 2024-01-11T21:49:56.161 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 1% complete... Thick provisioning: 2% complete... Thick provisioning: 3% complete... Thick provisioning: 4% complete...2024-01-11T21:49:56.159+0000 7f5d245c4700 0 -- 172.21.15.178:0/4015753318 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x5570b7c215e0 msgr2=0x5570b7c23a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:49:56.243 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 5% complete...2024-01-11T21:49:56.239+0000 7f5d24dc5700 0 -- 172.21.15.178:0/4015753318 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7f5d04037010 msgr2=0x7f5d04039430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:50:03.850 INFO:tasks.workunit.client.0.smithi178.stderr: Thick provisioning: 6% complete... Thick provisioning: 7% complete... Thick provisioning: 8% complete... Thick provisioning: 9% complete... Thick provisioning: 10% complete... Thick provisioning: 11% complete... Thick provisioning: 12% complete... Thick provisioning: 13% complete... Thick provisioning: 14% complete... Thick provisioning: 15% complete... Thick provisioning: 16% complete... Thick provisioning: 17% complete... Thick provisioning: 18% complete... Thick provisioning: 19% complete... Thick provisioning: 20% complete... Thick provisioning: 21% complete... Thick provisioning: 22% complete... Thick provisioning: 23% complete... Thick provisioning: 24% complete... Thick provisioning: 25% complete... Thick provisioning: 26% complete... Thick provisioning: 27% complete... Thick provisioning: 28% complete... Thick provisioning: 29% complete... Thick provisioning: 30% complete... Thick provisioning: 31% complete... Thick provisioning: 32% complete... Thick provisioning: 33% complete... Thick provisioning: 34% complete... Thick provisioning: 35% complete... Thick provisioning: 36% complete... Thick provisioning: 37% complete... Thick provisioning: 38% complete... Thick provisioning: 39% complete... Thick provisioning: 40% complete... Thick provisioning: 41% complete... Thick provisioning: 42% complete... Thick provisioning: 43% complete... Thick provisioning: 44% complete... Thick provisioning: 45% complete... Thick provisioning: 46% complete... Thick provisioning: 47% complete... Thick provisioning: 48% complete... Thick provisioning: 49% complete... Thick provisioning: 50% complete... Thick provisioning: 51% complete... Thick provisioning: 52% complete... Thick provisioning: 53% complete... Thick provisioning: 54% complete... Thick provisioning: 55% complete... Thick provisioning: 56% complete... Thick provisioning: 57% complete... Thick provisioning: 58% complete... Thick provisioning: 59% complete... Thick provisioning: 60% complete... Thick provisioning: 61% complete... Thick provisioning: 62% complete... Thick provisioning: 63% complete... Thick provisioning: 64% complete... Thick provisioning: 65% complete... Thick provisioning: 66% complete... Thick provisioning: 67% complete... Thick provisioning: 68% complete... Thick provisioning: 69% complete... Thick provisioning: 70% complete... Thick provisioning: 71% complete... Thick provisioning: 72% complete... Thick provisioning: 73% complete... Thick provisioning: 74% complete... Thick provisioning: 75% complete... Thick provisioning: 76% complete... Thick provisioning: 77% complete... Thick provisioning: 78% complete... Thick provisioning: 79% complete... Thick provisioning: 80% complete... Thick provisioning: 81% complete... Thick provisioning: 82% complete... Thick provisioning: 83% complete... Thick provisioning: 84% complete... Thick provisioning: 85% complete... Thick provisioning: 86% complete... Thick provisioning: 87% complete... Thick provisioning: 88% complete... Thick provisioning: 89% complete... Thick provisioning: 90% complete... Thick provisioning: 91% complete... Thick provisioning: 92% complete... Thick provisioning: 93% complete... Thick provisioning: 94% complete... Thick provisioning: 95% complete... Thick provisioning: 96% complete... Thick provisioning: 97% complete... Thick provisioning: 98% complete... Thick provisioning: 99% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2024-01-11T21:50:03.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ count=0 2024-01-11T21:50:03.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret= 2024-01-11T21:50:03.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 -lt 10 ']' 2024-01-11T21:50:03.867 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:50:03.868 INFO:tasks.workunit.client.0.smithi178.stderr:+ tr -s ' ' 2024-01-11T21:50:03.868 INFO:tasks.workunit.client.0.smithi178.stderr:+ cut -d ' ' -f 4-5 2024-01-11T21:50:03.868 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:50:03.868 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^4 GiB' 2024-01-11T21:50:03.921 INFO:tasks.workunit.client.0.smithi178.stdout:4 GiB 2024-01-11T21:50:03.922 INFO:tasks.workunit.client.0.smithi178.stderr:+ ret=0 2024-01-11T21:50:03.923 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 = 0 ']' 2024-01-11T21:50:03.923 INFO:tasks.workunit.client.0.smithi178.stderr:+ break 2024-01-11T21:50:03.923 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd du 2024-01-11T21:50:03.971 INFO:tasks.workunit.client.0.smithi178.stdout:NAME PROVISIONED USED 2024-01-11T21:50:03.972 INFO:tasks.workunit.client.0.smithi178.stdout:test1 4 GiB 4 GiB 2024-01-11T21:50:03.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ '[' 0 '!=' 0 ']' 2024-01-11T21:50:03.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd rm test1 2024-01-11T21:50:04.210 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 1% complete... Removing image: 2% complete...2024-01-11T21:50:04.207+0000 7faa2cf90700 0 -- 172.21.15.178:0/1599904197 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x56270b5e8020 msgr2=0x56270b688bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:50:04.239 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 3% complete...2024-01-11T21:50:04.235+0000 7faa2cf90700 0 -- 172.21.15.178:0/1599904197 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7faa0c036f40 msgr2=0x7faa0c039360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:50:05.256 INFO:tasks.workunit.client.0.smithi178.stderr: Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-01-11T21:50:05.260 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd ls 2024-01-11T21:50:05.260 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test1 2024-01-11T21:50:05.260 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:50:05.261 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:50:05.296 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:50:05.297 INFO:tasks.workunit.client.0.smithi178.stderr:+ test_namespace 2024-01-11T21:50:05.297 INFO:tasks.workunit.client.0.smithi178.stderr:+ echo 'testing namespace...' 2024-01-11T21:50:05.297 INFO:tasks.workunit.client.0.smithi178.stdout:testing namespace... 2024-01-11T21:50:05.298 INFO:tasks.workunit.client.0.smithi178.stderr:+ remove_images 2024-01-11T21:50:05.298 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:05.425 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:05.542 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:05.675 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:05.814 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:05.960 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:06.108 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:06.247 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:06.391 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:06.533 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:06.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:06.827 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:06.971 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:07.104 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:07.245 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:07.393 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:07.537 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:07.680 INFO:tasks.workunit.client.0.smithi178.stderr:+ for img in $IMGS 2024-01-11T21:50:07.830 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:50:07.830 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^0$' 2024-01-11T21:50:07.830 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace ls 2024-01-11T21:50:07.872 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-01-11T21:50:07.873 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace create rbd/test1 2024-01-11T21:50:07.922 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace create --pool rbd --namespace test2 2024-01-11T21:50:07.975 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace create --namespace test3 2024-01-11T21:50:08.028 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd namespace create rbd/test3 2024-01-11T21:50:08.028 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace create rbd/test3 2024-01-11T21:50:08.071 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: failed to created namespace: 2024-01-11T21:50:08.067+0000 7f2ff3b4e4c0 -1 librbd::api::Namespace: create: failed to add namespace: (17) File exists 2024-01-11T21:50:08.071 INFO:tasks.workunit.client.0.smithi178.stderr:(17) File exists 2024-01-11T21:50:08.075 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:50:08.076 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace list 2024-01-11T21:50:08.076 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep test 2024-01-11T21:50:08.076 INFO:tasks.workunit.client.0.smithi178.stderr:+ wc -l 2024-01-11T21:50:08.076 INFO:tasks.workunit.client.0.smithi178.stderr:+ grep '^3$' 2024-01-11T21:50:08.115 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-01-11T21:50:08.116 INFO:tasks.workunit.client.0.smithi178.stderr:+ expect_fail rbd namespace remove --pool rbd missing 2024-01-11T21:50:08.116 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd namespace remove --pool rbd missing 2024-01-11T21:50:08.142 INFO:tasks.workunit.client.0.smithi178.stderr:rbd: namespace name was not specified 2024-01-11T21:50:08.144 INFO:tasks.workunit.client.0.smithi178.stderr:+ return 0 2024-01-11T21:50:08.144 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd create --image-format 2 --size 1G rbd/test1/image1 2024-01-11T21:50:08.230 INFO:tasks.workunit.client.0.smithi178.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 32M --io-size 4K rbd/test1/image1 2024-01-11T21:50:08.305 INFO:tasks.workunit.client.0.smithi178.stdout:bench type write io_size 4096 io_threads 16 bytes 33554432 pattern random 2024-01-11T21:50:09.421 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:50:09.419+0000 7fd5e8fc6700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x560328be4050 msgr2=0x7fd5cc060cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T21:50:09.580 INFO:tasks.workunit.client.0.smithi178.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-01-11T21:50:09.580 INFO:tasks.workunit.client.0.smithi178.stdout: 1 3264 2586.81 10 MiB/s 2024-01-11T21:50:10.791 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T21:50:10.787+0000 7fd5e97c7700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x7fd5cc036fe0 msgr2=0x7fd5cc039400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-01-11T21:50:11.012 INFO:tasks.workunit.client.0.smithi178.stdout: 2 3440 1280.03 5.0 MiB/s 2024-01-11T21:50:41.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:41.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.10580.0:732 3.0 3:3aa1bbfb:test1::rbd_data.2951ff622ce2.0000000000000065:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2736128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:42.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:42.218+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:42.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:42.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:43.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:43.246+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:43.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:43.730+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:44.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:44.266+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:44.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:44.774+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:45.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:45.306+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:45.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:45.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:46.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:46.302+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:46.750 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:46.746+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:47.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:47.314+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:47.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:47.750+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:48.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:48.342+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:48.745 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:48.742+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:49.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:49.362+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:49.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:49.754+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:50.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:50.322+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:50.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:50.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:51.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:51.362+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:51.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:51.782+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:52.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:52.382+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:52.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:52.762+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:53.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:53.430+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:53.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:53.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:54.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:54.414+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:54.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:54.766+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:55.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:55.454+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:55.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:55.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:56.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:56.474+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:56.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:56.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:57.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:57.474+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:57.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:57.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:58.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:58.470+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:58.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:58.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:59.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:50:59.454+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:50:59.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:50:59.750+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:00.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:00.434+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:00.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:00.702+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:01.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:01.470+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:01.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:01.714+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:02.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:02.518+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:02.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:02.738+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:03.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:03.498+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:03.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:03.750+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:04.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:04.486+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:04.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:04.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:05.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:05.446+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:05.792 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:05.786+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:06.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:06.402+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:06.810 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:06.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:07.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:07.430+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:07.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:07.766+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:08.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:08.438+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:08.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:08.734+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:09.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:09.454+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:09.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:09.690+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:10.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:10.418+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:10.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:10.690+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:11.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:11.418+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:11.684 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:11.678+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:12.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:12.386+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:12.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:12.658+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:13.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:13.357+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:13.675 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:13.669+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:14.403 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:14.397+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:14.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:14.677+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:15.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:15.405+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:15.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:15.685+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:16.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:16.385+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:16.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:16.649+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:17.402 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:17.397+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:17.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:17.677+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:18.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:18.425+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:18.663 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:18.657+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:19.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:19.397+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:19.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:19.637+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:20.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:20.405+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:20.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:20.593+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:21.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:21.457+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:21.564 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:21.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:22.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:22.469+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:22.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:22.569+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:23.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:23.493+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:23.619 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:23.613+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:24.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:24.461+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:24.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:24.589+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:25.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:25.493+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:25.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:25.637+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:26.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:26.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:26.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:26.641+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:27.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:27.541+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:27.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:27.693+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:28.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:28.517+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:28.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:28.709+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:29.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:29.509+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:29.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:29.701+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:30.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:30.529+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:30.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:30.749+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:31.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:31.489+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:31.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:31.713+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:32.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:32.465+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:32.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:32.689+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:33.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:33.485+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:33.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:33.689+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:34.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:34.493+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:34.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:34.713+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:35.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:35.529+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:35.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:35.753+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:36.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:36.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:36.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:36.753+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:37.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:37.541+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:37.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:37.757+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:38.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:38.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:38.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:38.733+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:39.573 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:39.569+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:39.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:39.777+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:40.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:40.585+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:40.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:40.765+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:41.593 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:41.585+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:41.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:41.725+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:42.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:42.565+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:42.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:42.701+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:43.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:43.557+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:43.663 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:43.657+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:44.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:44.517+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:44.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:44.629+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:45.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:45.489+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:45.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:45.621+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:46.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:46.441+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:46.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:46.585+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:47.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:47.453+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:47.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:47.553+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:48.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:48.481+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:48.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:48.601+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:49.504 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:49.497+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:49.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:49.577+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:50.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:50.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:50.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:50.625+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:51.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:51.473+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:51.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:51.637+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:52.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:52.505+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:52.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:52.625+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:53.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:53.465+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:53.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:53.649+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:54.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:54.453+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:54.617 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:54.609+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:55.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:55.501+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:55.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:55.589+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:56.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:56.497+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:56.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:56.637+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:57.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:57.488+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:57.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:57.648+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:58.458 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:58.452+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:58.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:58.624+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:59.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:51:59.480+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:51:59.675 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:51:59.668+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:00.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:00.528+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:00.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:00.712+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:01.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:01.504+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:01.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:01.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:02.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:02.544+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:02.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:02.728+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:03.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:03.552+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:03.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:03.772+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:04.550 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:04.544+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:04.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:04.732+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:05.587 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:05.580+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:05.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:05.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:06.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:06.592+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:06.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:06.720+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:07.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:07.636+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:07.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:07.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:08.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:08.644+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:08.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:08.724+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:09.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:09.636+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:09.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:09.740+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:10.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:10.676+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:10.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:10.700+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:11.667 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:11.660+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:11.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:11.712+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:12.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:12.668+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:12.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:12.676+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:13.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:13.636+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:13.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:13.672+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:14.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:14.648+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:14.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:14.652+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:15.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:15.652+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:15.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:15.696+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:16.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:16.632+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:16.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:16.684+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:17.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:17.616+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:17.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:17.720+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:18.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:18.576+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:18.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:18.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:19.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:19.548+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:19.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:19.740+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:20.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:20.552+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:20.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:20.712+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:21.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:21.520+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:21.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:21.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:22.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:22.476+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:22.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:22.676+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:23.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:23.428+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:23.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:23.708+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:24.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:24.444+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:24.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:24.712+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:25.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:25.404+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:25.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:25.708+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:26.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:26.420+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:26.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:26.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:27.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:27.380+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:27.667 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:27.660+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:28.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:28.400+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:28.667 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:28.660+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:29.369 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:29.360+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:29.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:29.700+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:30.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:30.320+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:30.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:30.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:31.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:31.312+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:31.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:31.688+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:32.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:32.316+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:32.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:32.712+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:33.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:33.308+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:33.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:33.736+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:34.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:34.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:34.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:34.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:35.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:35.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:35.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:35.728+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:36.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:36.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:36.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:36.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:37.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:37.300+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:37.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:37.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:38.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:38.268+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:38.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:38.752+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:39.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:39.300+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:39.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:39.724+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:40.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:40.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:40.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:40.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:41.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:41.368+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:41.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:41.668+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:42.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:42.344+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:42.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:42.628+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:43.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:43.384+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:43.606 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:43.604+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:44.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:44.364+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:44.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:44.556+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:45.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:45.396+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:45.593 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:45.588+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:46.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:46.364+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:46.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:46.600+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:47.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:47.408+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:47.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:47.648+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:48.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:48.400+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:48.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:48.632+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:49.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:49.404+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:49.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:49.628+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:50.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:50.359+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:50.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:50.611+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:51.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:51.351+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:51.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:51.655+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:52.377 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:52.371+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:52.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:52.699+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:53.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:53.363+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:53.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:53.723+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:54.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:54.391+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:54.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:54.711+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:55.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:55.443+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:55.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:55.711+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:56.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:56.411+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:56.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:56.707+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:57.377 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:57.371+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:57.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:57.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:58.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:58.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:58.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:58.771+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:59.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:52:59.327+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:52:59.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:52:59.775+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:00.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:00.315+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:00.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:00.751+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:01.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:01.299+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:01.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:01.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:02.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:02.263+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:02.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:02.727+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:03.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:03.291+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:03.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:03.711+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:04.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:04.291+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:04.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:04.707+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:05.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:05.311+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:05.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:05.679+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:06.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:06.263+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:06.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:06.683+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:07.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:07.235+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:07.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:07.691+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:08.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:08.219+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:08.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:08.647+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:09.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:09.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:09.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:09.687+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:10.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:10.251+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:10.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:10.639+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:11.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:11.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:11.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:11.683+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:12.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:12.215+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:12.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:12.651+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:13.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:13.219+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:13.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:13.635+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:14.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:14.199+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:14.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:14.627+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:15.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:15.159+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:15.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:15.607+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:16.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:16.131+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:16.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:16.571+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:17.176 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:17.171+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:17.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:17.527+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:18.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:18.207+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:18.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:18.495+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:19.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:19.167+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:19.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:19.487+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:20.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:20.119+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:20.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:20.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:21.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:21.139+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:21.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:21.435+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:22.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:22.119+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:22.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:22.391+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:23.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:23.091+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:23.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:23.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:24.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:24.115+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:24.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:24.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:25.099 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:25.095+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:25.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:25.439+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:26.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:26.099+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:26.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:26.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:27.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:27.079+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:27.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:27.455+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:28.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:28.119+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:28.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:28.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:29.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:29.127+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:29.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:29.443+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:30.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:30.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:30.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:30.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:31.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:31.187+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:31.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:31.367+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:32.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:32.159+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:32.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:32.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:33.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:33.127+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:33.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:33.435+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:34.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:34.079+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:34.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:34.411+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:35.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:35.043+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:35.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:35.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:36.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:36.055+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:36.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:36.419+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:37.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:37.091+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:37.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:37.471+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:38.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:38.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:38.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:38.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:39.060 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:39.055+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:39.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:39.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:40.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:40.071+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:40.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:40.475+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:41.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:41.031+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:41.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:41.447+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:42.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:42.038+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:42.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:42.446+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:43.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:43.062+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:43.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:43.474+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:44.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:44.014+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:44.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:44.434+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:45.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:45.030+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:45.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:45.422+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:45.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:45.994+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:46.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:46.438+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:47.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:47.038+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:47.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:47.470+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:48.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:48.054+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:48.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:48.446+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:49.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:49.050+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:49.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:49.446+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:50.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:50.094+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:50.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:50.438+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:51.110 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:51.102+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:51.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:51.486+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:52.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:52.142+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:52.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:52.450+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:53.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:53.094+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:53.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:53.454+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:54.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:54.110+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:54.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:54.494+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:55.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:55.074+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:55.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:55.518+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:56.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:56.078+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:56.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:56.478+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:57.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:57.114+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:57.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:57.474+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:58.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:58.130+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:58.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:58.458+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:59.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:53:59.166+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:53:59.499 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:53:59.494+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:00.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:00.174+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:00.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:00.542+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:01.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:01.154+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:01.563 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:01.558+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:02.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:02.190+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:02.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:02.602+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:03.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:03.230+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:03.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:03.610+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:04.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:04.210+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:04.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:04.562+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:05.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:05.258+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:05.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:05.610+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:06.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:06.282+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:06.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:06.574+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:07.274 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:07.266+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:07.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:07.590+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:08.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:08.282+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:08.591 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:08.586+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:09.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:09.306+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:09.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:09.614+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:10.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:10.262+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:10.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:10.622+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:11.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:11.246+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:11.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:11.602+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:12.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:12.226+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:12.562 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:12.558+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:13.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:13.238+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:13.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:13.530+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:14.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:14.246+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:14.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:14.534+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:15.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:15.206+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:15.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:15.554+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:16.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:16.178+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:16.578 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:16.570+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:17.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:17.162+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:17.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:17.610+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:18.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:18.174+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:18.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:18.590+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:19.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:19.146+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:19.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:19.630+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:20.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:20.110+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:20.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:20.626+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:21.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:21.138+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:21.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:21.654+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:22.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:22.182+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:22.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:22.666+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:23.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:23.222+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:23.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:23.626+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:24.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:24.210+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:24.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:24.618+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:25.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:25.258+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:25.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:25.574+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:26.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:26.242+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:26.584 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:26.578+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:27.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:27.226+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:27.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:27.586+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:28.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:28.218+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:28.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:28.610+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:29.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:29.234+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:29.584 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:29.578+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:30.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:30.206+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:30.562 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:30.558+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:31.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:31.226+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:31.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:31.533+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:32.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:32.213+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:32.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:32.569+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:33.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:33.253+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:33.569 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:33.561+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:34.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:34.225+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:34.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:34.613+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:35.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:35.229+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:35.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:35.601+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:36.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:36.237+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:36.612 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:36.605+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:37.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:37.253+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:37.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:37.653+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:38.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:38.273+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:38.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:38.653+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:39.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:39.233+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:39.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:39.673+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:40.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:40.253+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:40.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:40.685+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:41.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:41.281+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:41.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:41.697+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:42.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:42.293+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:42.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:42.725+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:43.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:43.285+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:43.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:43.741+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:44.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:44.313+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:44.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:44.725+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:45.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:45.341+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:45.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:45.725+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:46.361 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:46.353+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:46.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:46.757+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:47.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:47.349+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:47.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:47.733+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:48.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:48.389+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:48.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:48.761+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:49.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:49.433+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:49.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:49.801+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:50.405 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:50.397+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:50.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:50.801+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:51.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:51.437+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:51.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:51.849+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:52.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:52.445+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:52.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:52.805+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:53.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:53.401+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:53.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:53.849+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:54.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:54.409+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:54.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:54.877+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:55.466 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:55.461+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:55.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:55.913+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:56.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:56.489+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:56.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:56.873+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:57.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:57.513+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:57.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:57.921+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:58.554 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:58.549+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:58.968 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:58.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:59.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:54:59.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:54:59.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:54:59.985+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:00.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:00.553+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:01.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:01.001+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:01.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:01.557+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:01.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:01.953+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:02.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:02.605+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:02.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:02.925+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:03.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:03.605+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:03.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:03.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:04.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:04.629+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:04.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:04.973+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:05.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:05.609+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:05.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:05.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:06.598 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:06.593+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:06.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:06.965+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:07.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:07.597+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:07.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:07.993+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:08.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:08.569+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:09.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:09.017+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:09.543 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:09.537+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:10.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:10.017+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:10.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:10.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:10.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:10.985+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:11.544 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:11.537+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:12.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:12.025+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:12.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:12.501+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:13.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:13.021+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:13.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:13.513+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:14.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:14.065+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:14.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:14.537+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:15.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:15.029+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:15.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:15.513+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:16.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:16.001+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:16.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:16.489+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:16.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:16.973+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:17.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:17.517+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:17.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:17.985+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:18.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:18.545+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:19.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:19.005+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:19.544 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:19.537+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:19.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:19.985+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:20.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:20.504+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:20.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:20.980+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:21.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:21.544+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:22.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:21.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:22.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:22.544+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:23.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:23.028+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:23.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:23.576+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:24.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:24.068+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:24.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:24.592+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:25.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:25.064+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:25.571 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:25.564+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:26.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:26.028+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:26.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:26.520+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:27.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:27.012+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:27.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:27.564+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:27.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:27.972+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:28.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:28.516+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:29.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:29.024+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:29.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:29.560+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:30.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:30.072+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:30.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:30.576+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:31.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:31.040+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:31.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:31.540+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:32.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:32.000+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:32.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:32.532+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:33.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:33.036+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:33.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:33.572+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:34.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:33.992+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:34.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:34.604+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:35.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:34.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:35.587 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:35.580+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:36.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:36.008+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:36.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:36.548+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:37.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:37.020+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:37.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:37.504+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:38.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:38.032+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:38.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:38.488+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:39.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:39.028+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:39.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:39.512+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:39.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:39.988+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:40.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:40.464+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:40.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:40.968+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:41.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:41.488+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:42.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:42.012+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:42.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:42.520+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:42.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:42.976+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:43.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:43.504+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:43.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:43.960+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:44.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:44.508+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:45.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:44.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:45.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:45.460+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:45.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:45.956+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:46.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:46.492+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:46.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:46.932+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:47.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:47.528+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:47.962 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:47.952+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:48.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:48.504+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:48.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:48.972+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:49.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:49.484+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:49.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:49.972+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:50.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:50.460+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:50.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:50.948+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:51.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:51.424+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:52.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:51.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:52.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:52.416+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:53.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:53.044+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:53.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:53.416+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:54.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:54.072+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:54.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:54.424+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:55.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:55.048+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:55.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:55.396+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:56.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:56.032+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:56.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:56.384+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:57.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:57.052+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:57.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:57.368+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:58.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:58.040+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:58.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:58.408+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:59.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:55:59.060+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:55:59.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:55:59.456+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:00.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:00.060+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:00.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:00.412+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:01.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:01.044+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:01.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:01.456+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:02.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:02.008+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:02.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:02.508+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:02.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:02.960+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:03.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:03.516+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:04.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:04.012+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:04.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:04.540+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:05.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:05.000+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:05.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:05.508+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:05.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:05.956+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:06.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:06.492+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:06.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:06.988+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:07.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:07.524+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:07.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:07.963+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:08.555 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:08.551+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:08.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:08.947+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:09.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:09.535+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:09.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:09.919+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:10.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:10.491+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:10.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:10.967+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:11.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:11.531+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:11.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:11.971+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:12.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:12.567+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:13.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:13.003+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:13.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:13.611+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:13.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:13.971+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:14.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:14.643+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:15.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:15.011+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:15.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:15.683+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:16.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:15.999+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:16.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:16.639+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:17.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:17.007+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:17.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:17.655+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:17.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:17.975+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:18.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:18.615+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:18.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:18.987+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:19.605 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:19.599+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:19.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:19.951+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:20.640 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:20.635+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:20.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:20.919+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:21.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:21.667+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:21.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:21.951+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:22.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:22.619+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:22.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:22.931+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:23.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:23.643+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:23.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:23.895+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:24.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:24.679+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:24.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:24.855+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:25.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:25.687+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:25.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:25.871+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:26.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:26.735+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:26.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:26.903+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:27.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:27.759+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:27.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:27.911+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:28.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:28.807+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:28.873 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:28.867+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:29.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:29.847+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:29.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:29.895+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:30.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:30.855+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:30.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:30.863+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:31.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:31.819+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:31.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:31.851+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:32.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:32.771+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:32.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:32.859+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:33.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:33.747+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:33.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:33.855+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:34.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:34.779+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:34.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:34.807+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:35.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:35.775+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:35.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:35.855+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:36.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:36.763+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:36.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:36.875+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:37.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:37.807+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:37.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:37.899+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:38.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:38.791+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:38.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:38.891+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:39.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:39.827+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:39.895 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:39.891+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:40.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:40.839+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:40.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:40.915+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:41.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:41.839+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:41.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:41.967+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:42.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:42.807+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:43.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:43.007+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:43.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:43.787+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:44.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:44.051+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:44.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:44.787+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:45.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:45.003+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:45.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:45.815+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:45.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:45.963+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:46.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:46.851+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:46.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:46.923+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:47.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:47.867+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:47.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:47.943+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:48.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:48.867+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:48.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:48.971+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:49.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:49.915+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:50.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:49.995+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:50.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:50.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:51.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:51.007+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:51.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:51.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:51.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:51.991+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:52.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:52.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:52.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:52.951+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:53.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:53.847+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:53.914 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:53.911+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:54.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:54.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:54.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:54.923+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:55.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:55.862+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:55.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:55.874+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:56.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:56.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:56.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:56.874+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:57.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:57.858+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:57.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:57.866+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:58.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:58.846+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:58.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:58.874+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:59.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:56:59.830+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:56:59.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:56:59.878+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:00.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:00.874+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:00.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:00.890+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:01.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:01.898+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:01.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:01.910+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:02.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:02.862+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:02.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:02.898+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:03.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:03.818+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:03.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:03.850+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:04.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:04.770+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:04.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:04.834+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:05.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:05.786+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:05.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:05.854+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:06.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:06.738+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:06.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:06.870+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:07.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:07.746+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:07.926 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:07.922+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:08.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:08.726+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:08.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:08.962+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:09.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:09.738+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:09.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:09.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:10.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:10.758+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:10.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:10.942+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:11.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:11.730+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:11.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:11.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:12.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:12.702+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:13.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:13.010+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:13.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:13.746+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:13.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:13.978+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:14.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:14.774+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:14.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:14.982+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:15.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:15.810+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:15.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:15.990+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:16.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:16.826+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:16.952 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:16.946+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:17.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:17.790+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:17.933 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:17.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:18.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:18.826+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:18.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:18.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:19.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:19.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:19.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:19.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:20.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:20.810+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:20.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:20.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:21.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:21.782+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:21.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:21.962+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:22.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:22.814+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:22.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:22.962+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:23.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:23.790+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:23.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:23.914+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:24.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:24.834+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:24.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:24.898+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:25.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:25.858+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:25.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:25.914+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:26.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:26.898+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:26.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:26.954+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:27.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:27.926+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:27.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:27.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:28.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:28.942+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:28.952 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:28.946+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:29.967 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:29.962+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:29.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:29.962+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:30.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:30.942+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:31.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:30.998+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:31.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:31.898+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:32.041 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:32.034+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:32.949 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:32.942+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:33.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:33.002+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:33.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:33.918+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:33.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:33.978+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:34.908 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:34.902+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:34.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:34.994+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:35.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:35.922+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:35.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:35.990+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:36.941 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:36.934+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:37.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:37.042+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:37.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:37.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:38.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:38.014+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:38.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:38.898+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:39.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:39.058+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:39.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:39.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:40.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:40.090+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:40.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:40.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:41.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:41.062+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:41.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:41.929+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:42.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:42.061+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:42.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:42.905+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:43.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:43.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:43.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:43.945+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:44.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:44.089+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:44.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:44.937+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:45.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:45.077+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:45.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:45.917+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:46.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:46.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:46.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:46.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:47.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:47.021+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:47.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:47.937+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:48.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:48.021+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:48.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:48.985+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:49.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:49.037+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:50.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:50.005+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:50.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:50.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:51.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:50.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:51.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:51.045+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:51.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:51.977+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:52.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:52.057+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:53.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:52.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:53.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:53.041+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:53.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:53.969+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:54.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:54.041+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:55.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:55.009+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:55.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:55.093+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:56.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:56.029+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:56.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:56.061+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:57.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:57.041+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:57.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:57.073+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:58.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:57.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:58.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:58.077+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:59.023 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:57:59.017+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:57:59.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:57:59.101+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:00.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:00.037+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:00.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:00.053+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:01.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:01.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:01.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:01.077+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:02.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:02.089+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:02.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:02.105+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:03.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:03.093+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:03.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:03.109+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:04.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:04.089+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:04.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:04.117+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:05.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:05.069+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:05.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:05.121+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:06.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:06.021+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:06.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:06.137+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:07.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:07.021+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:07.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:07.189+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:08.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:08.013+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:08.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:08.169+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:08.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:08.973+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:09.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:09.213+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:10.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:10.017+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:10.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:10.189+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:11.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:11.001+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:11.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:11.213+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:12.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:12.037+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:12.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:12.237+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:13.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:13.065+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:13.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:13.289+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:14.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:14.045+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:14.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:14.329+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:15.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:15.097+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:15.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:15.341+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:16.070 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:16.061+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:16.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:16.369+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:17.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:17.113+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:17.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:17.409+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:18.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:18.117+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:18.414 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:18.409+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:19.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:19.121+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:19.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:19.433+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:20.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:20.169+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:20.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:20.421+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:21.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:21.197+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:21.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:21.445+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:22.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:22.149+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:22.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:22.469+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:23.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:23.129+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:23.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:23.509+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:24.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:24.133+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:24.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:24.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:25.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:25.113+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:25.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:25.481+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:26.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:26.105+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:26.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:26.481+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:27.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:27.149+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:27.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:27.465+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:28.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:28.120+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:28.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:28.448+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:29.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:29.156+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:29.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:29.452+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:30.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:30.172+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:30.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:30.412+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:31.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:31.132+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:31.403 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:31.396+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:32.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:32.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:32.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:32.380+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:33.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:33.152+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:33.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:33.396+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:34.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:34.104+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:34.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:34.424+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:35.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:35.108+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:35.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:35.420+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:36.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:36.148+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:36.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:36.372+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:37.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:37.176+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:37.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:37.344+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:38.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:38.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:38.338 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:38.328+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:39.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:39.152+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:39.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:39.380+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:40.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:40.156+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:40.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:40.344+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:41.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:41.136+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:41.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:41.324+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:42.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:42.184+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:42.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:42.352+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:43.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:43.204+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:43.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:43.324+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:44.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:44.180+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:44.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:44.308+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:45.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:45.204+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:45.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:45.324+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:46.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:46.168+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:46.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:46.276+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:47.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:47.188+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:47.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:47.260+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:48.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:48.228+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:48.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:48.228+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:49.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:49.200+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:49.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:49.228+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:50.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:50.168+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:50.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:50.212+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:51.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:51.132+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:51.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:51.208+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:52.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:52.124+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:52.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:52.168+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:53.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:53.164+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:53.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:53.204+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:54.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:54.184+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:54.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:54.216+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:55.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:55.148+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:55.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:55.208+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:56.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:56.132+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:56.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:56.228+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:57.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:57.152+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:57.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:57.212+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:58.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:58.164+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:58.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:58.184+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:59.137 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:58:59.128+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:58:59.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:58:59.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:00.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:00.156+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:00.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:00.180+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:01.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:01.176+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:01.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:01.232+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:02.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:02.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:02.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:02.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:03.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:03.268+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:03.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:03.312+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:04.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:04.264+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:04.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:04.332+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:05.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:05.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:05.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:05.372+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:06.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:06.276+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:06.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:06.360+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:07.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:07.248+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:07.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:07.376+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:08.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:08.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:08.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:08.384+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:09.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:09.256+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:09.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:09.424+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:10.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:10.232+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:10.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:10.392+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:11.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:11.264+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:11.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:11.356+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:12.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:12.232+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:12.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:12.364+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:13.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:13.260+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:13.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:13.412+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:14.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:14.215+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:14.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:14.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:15.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:15.211+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:15.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:15.383+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:16.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:16.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:16.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:16.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:17.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:17.131+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:17.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:17.371+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:18.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:18.123+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:18.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:18.347+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:19.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:19.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:19.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:19.311+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:20.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:20.115+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:20.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:20.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:21.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:21.135+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:21.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:21.379+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:22.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:22.179+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:22.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:22.411+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:23.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:23.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:23.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:23.439+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:24.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:24.207+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:24.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:24.475+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:25.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:25.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:25.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:25.455+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:26.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:26.203+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:26.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:26.491+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:27.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:27.203+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:27.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:27.515+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:28.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:28.227+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:28.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:28.507+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:29.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:29.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:29.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:29.547+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:30.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:30.279+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:30.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:30.595+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:31.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:31.267+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:31.562 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:31.555+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:32.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:32.315+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:32.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:32.527+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:33.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:33.355+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:33.565 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:33.559+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:34.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:34.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:34.593 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:34.587+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:35.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:35.383+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:35.617 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:35.611+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:36.379 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:36.375+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:36.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:36.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:37.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:37.391+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:37.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:37.523+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:38.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:38.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:38.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:38.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:39.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:39.367+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:39.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:39.507+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:40.379 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:40.375+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:40.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:40.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:41.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:41.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:41.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:41.471+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:42.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:42.363+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:42.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:42.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:43.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:43.351+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:43.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:43.479+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:44.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:44.339+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:44.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:44.463+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:45.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:45.327+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:45.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:45.475+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:46.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:46.355+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:46.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:46.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:47.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:47.363+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:47.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:47.483+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:48.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:48.315+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:48.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:48.471+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:49.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:49.307+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:49.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:49.503+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:50.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:50.263+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:50.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:50.491+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:51.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:51.307+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:51.545 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:51.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:52.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:52.267+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:52.553 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:52.547+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:53.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:53.299+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:53.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:53.523+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:54.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:54.295+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:54.543 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:54.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:55.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:55.311+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:55.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:55.495+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:56.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:56.287+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:56.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:56.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:57.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:57.311+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:57.565 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:57.559+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:58.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:58.307+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:58.596 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:58.591+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:59.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T21:59:59.258+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T21:59:59.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T21:59:59.598+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:00.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:00.290+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:00.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:00.622+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:01.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:01.306+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:01.675 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:01.670+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:02.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:02.326+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:02.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:02.654+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:03.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:03.358+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:03.613 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:03.606+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:04.403 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:04.398+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:04.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:04.642+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:05.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:05.398+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:05.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:05.650+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:06.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:06.366+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:06.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:06.606+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:07.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:07.406+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:07.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:07.622+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:08.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:08.386+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:08.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:08.634+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:09.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:09.422+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:09.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:09.622+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:10.381 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:10.374+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:10.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:10.662+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:11.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:11.366+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:11.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:11.702+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:12.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:12.414+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:12.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:12.722+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:13.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:13.378+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:13.752 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:13.746+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:14.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:14.370+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:14.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:14.734+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:15.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:15.370+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:15.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:15.730+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:16.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:16.390+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:16.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:16.766+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:17.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:17.374+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:17.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:17.762+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:18.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:18.374+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:18.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:18.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:19.335 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:19.330+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:19.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:19.790+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:20.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:20.330+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:20.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:20.834+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:21.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:21.374+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:21.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:21.846+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:22.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:22.370+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:22.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:22.826+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:23.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:23.362+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:23.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:23.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:24.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:24.358+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:24.815 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:24.810+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:25.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:25.362+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:25.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:25.850+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:26.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:26.378+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:26.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:26.826+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:27.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:27.410+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:27.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:27.790+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:28.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:28.418+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:28.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:28.746+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:29.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:29.406+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:29.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:29.702+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:30.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:30.454+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:30.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:30.694+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:31.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:31.410+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:31.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:31.698+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:32.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:32.442+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:32.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:32.678+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:33.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:33.426+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:33.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:33.678+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:34.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:34.454+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:34.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:34.722+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:35.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:35.490+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:35.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:35.742+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:36.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:36.510+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:36.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:36.702+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:37.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:37.470+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:37.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:37.754+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:38.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:38.430+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:38.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:38.766+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:39.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:39.466+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:39.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:39.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:40.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:40.498+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:40.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:40.746+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:41.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:41.494+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:41.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:41.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:42.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:42.450+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:42.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:42.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:43.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:43.406+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:43.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:43.750+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:44.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:44.426+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:44.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:44.746+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:45.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:45.441+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:45.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:45.709+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:46.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:46.437+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:46.675 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:46.669+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:47.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:47.389+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:47.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:47.717+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:48.402 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:48.393+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:48.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:48.689+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:49.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:49.345+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:49.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:49.729+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:50.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:50.329+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:50.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:50.777+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:51.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:51.313+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:51.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:51.761+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:52.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:52.289+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:52.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:52.729+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:53.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:53.301+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:53.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:53.745+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:54.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:54.321+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:54.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:54.741+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:55.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:55.321+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:55.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:55.769+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:56.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:56.317+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:56.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:56.781+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:57.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:57.317+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:57.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:57.745+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:58.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:58.321+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:58.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:58.765+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:59.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:00:59.273+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:00:59.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:00:59.721+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:00.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:00.241+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:00.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:00.677+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:01.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:01.221+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:01.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:01.717+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:02.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:02.233+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:02.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:02.685+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:03.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:03.197+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:03.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:03.665+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:04.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:04.153+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:04.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:04.713+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:05.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:05.201+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:05.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:05.729+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:06.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:06.193+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:06.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:06.745+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:07.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:07.233+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:07.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:07.765+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:08.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:08.189+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:08.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:08.813+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:09.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:09.193+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:09.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:09.857+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:10.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:10.169+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:10.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:10.861+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:11.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:11.133+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:11.917 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:11.909+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:12.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:12.169+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:12.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:12.897+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:13.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:13.153+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:13.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:13.901+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:14.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:14.121+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:14.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:14.953+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:15.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:15.161+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:15.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:15.981+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:16.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:16.145+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:16.968 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:16.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:17.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:17.173+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:18.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:18.005+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:18.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:18.197+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:19.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:19.033+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:19.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:19.173+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:20.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:20.017+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:20.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:20.133+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:21.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:21.009+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:21.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:21.165+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:22.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:22.033+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:22.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:22.169+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:23.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:23.025+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:23.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:23.145+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:24.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:24.009+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:24.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:24.193+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:24.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:24.981+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:25.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:25.189+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:26.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:26.029+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:26.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:26.241+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:27.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:26.997+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:27.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:27.221+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:28.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:28.041+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:28.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:28.261+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:29.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:29.089+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:29.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:29.229+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:30.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:30.077+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:30.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:30.269+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:31.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:31.085+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:31.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:31.237+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:32.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:32.073+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:32.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:32.281+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:33.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:33.085+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:33.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:33.277+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:34.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:34.105+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:34.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:34.285+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:35.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:35.073+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:35.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:35.253+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:36.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:36.077+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:36.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:36.245+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:37.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:37.041+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:37.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:37.201+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:38.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:38.081+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:38.224 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:38.217+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:39.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:39.129+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:39.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:39.221+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:40.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:40.117+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:40.266 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:40.261+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:41.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:41.161+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:41.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:41.249+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:42.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:42.129+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:42.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:42.269+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:43.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:43.093+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:43.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:43.289+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:44.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:44.045+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:44.340 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:44.333+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:45.059 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:45.053+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:45.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:45.317+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:46.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:46.025+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:46.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:46.281+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:47.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:47.061+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:47.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:47.301+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:48.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:48.069+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:48.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:48.285+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:49.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:49.117+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:49.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:49.309+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:50.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:50.069+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:50.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:50.269+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:51.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:51.053+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:51.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:51.309+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:52.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:52.041+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:52.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:52.353+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:53.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:53.069+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:53.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:53.345+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:54.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:54.029+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:54.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:54.313+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:55.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:55.001+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:55.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:55.321+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:56.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:56.045+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:56.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:56.373+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:57.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:57.009+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:57.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:57.401+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:58.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:58.029+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:58.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:58.440+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:59.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:01:59.008+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:01:59.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:01:59.416+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:00.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:00.056+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:00.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:00.384+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:01.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:01.096+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:01.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:01.388+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:02.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:02.060+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:02.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:02.352+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:03.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:03.040+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:03.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:03.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:04.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:04.056+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:04.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:04.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:05.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:05.084+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:05.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:05.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:06.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:06.064+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:06.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:06.324+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:07.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:07.056+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:07.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:07.288+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:08.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:08.096+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:08.267 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:08.260+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:09.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:09.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:09.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:09.236+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:10.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:10.080+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:10.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:10.276+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:11.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:11.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:11.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:11.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:12.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:12.096+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:12.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:12.252+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:13.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:13.080+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:13.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:13.280+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:14.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:14.116+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:14.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:14.288+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:15.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:15.120+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:15.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:15.288+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:16.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:16.080+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:16.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:16.252+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:17.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:17.096+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:17.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:17.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:18.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:18.116+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:18.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:18.284+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:19.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:19.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:19.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:19.280+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:20.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:20.148+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:20.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:20.312+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:21.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:21.104+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:21.337 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:21.328+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:22.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:22.100+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:22.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:22.312+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:23.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:23.092+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:23.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:23.304+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:24.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:24.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:24.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:24.300+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:25.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:25.092+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:25.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:25.280+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:26.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:26.108+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:26.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:26.268+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:27.083 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:27.076+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:27.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:27.228+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:28.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:28.028+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:28.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:28.192+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:29.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:29.040+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:29.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:29.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:30.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:29.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:30.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:30.216+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:30.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:30.988+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:31.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:31.184+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:31.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:31.988+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:32.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:32.208+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:33.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:33.036+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:33.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:33.196+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:34.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:34.044+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:34.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:34.172+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:35.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:35.084+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:35.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:35.196+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:36.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:36.084+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:36.172 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:36.164+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:37.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:37.048+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:37.126 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:37.120+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:38.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:38.076+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:38.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:38.168+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:39.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:39.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:39.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:39.184+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:40.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:40.176+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:40.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:40.216+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:41.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:41.132+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:41.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:41.216+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:42.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:42.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:42.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:42.252+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:43.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:43.100+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:43.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:43.232+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:44.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:44.132+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:44.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:44.248+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:45.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:45.172+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:45.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:45.220+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:46.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:46.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:46.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:46.232+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:47.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:47.100+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:47.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:47.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:48.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:48.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:48.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:48.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:49.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:49.148+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:49.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:49.228+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:50.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:50.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:50.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:50.244+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:51.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:51.164+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:51.267 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:51.264+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:52.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:52.200+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:52.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:52.252+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:53.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:53.236+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:53.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:53.248+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:54.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:54.256+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:54.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:54.256+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:55.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:55.264+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:55.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:55.284+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:56.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:56.240+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:56.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:56.252+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:57.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:57.280+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:57.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:57.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:58.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:58.260+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:58.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:58.292+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:59.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:02:59.304+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:02:59.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:02:59.312+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:00.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:00.312+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:00.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:00.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:01.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:01.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:01.361 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:01.356+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:02.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:02.292+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:02.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:02.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:03.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:03.307+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:03.385 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:03.379+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:04.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:04.331+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:04.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:04.359+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:05.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:05.335+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:05.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:05.379+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:06.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:06.323+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:06.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:06.335+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:07.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:07.347+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:07.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:07.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:08.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:08.363+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:08.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:08.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:09.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:09.335+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:09.423 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:09.419+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:10.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:10.359+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:10.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:10.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:11.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:11.331+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:11.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:11.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:12.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:12.323+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:12.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:12.411+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:13.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:13.351+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:13.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:13.431+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:14.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:14.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:14.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:14.435+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:15.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:15.323+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:15.426 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:15.423+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:16.337 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:16.331+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:16.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:16.431+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:17.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:17.375+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:17.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:17.439+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:18.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:18.363+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:18.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:18.439+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:19.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:19.395+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:19.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:19.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:20.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:20.351+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:20.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:20.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:21.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:21.383+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:21.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:21.419+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:22.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:22.387+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:22.413 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:22.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:23.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:23.359+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:23.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:23.431+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:24.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:24.347+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:24.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:24.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:25.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:25.303+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:25.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:25.499+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:26.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:26.299+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:26.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:26.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:27.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:27.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:27.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:27.455+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:28.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:28.247+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:28.506 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:28.499+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:29.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:29.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:29.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:29.515+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:30.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:30.207+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:30.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:30.551+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:31.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:31.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:31.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:31.507+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:32.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:32.271+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:32.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:32.527+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:33.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:33.319+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:33.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:33.479+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:34.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:34.287+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:34.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:34.531+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:35.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:35.323+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:35.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:35.563+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:36.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:36.287+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:36.535 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:36.531+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:37.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:37.291+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:37.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:37.579+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:38.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:38.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:38.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:38.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:39.274 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:39.271+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:39.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:39.579+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:40.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:40.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:40.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:40.571+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:41.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:41.283+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:41.562 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:41.555+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:42.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:42.331+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:42.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:42.591+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:43.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:43.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:43.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:43.631+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:44.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:44.311+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:44.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:44.623+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:45.358 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:45.355+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:45.632 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:45.627+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:46.405 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:46.399+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:46.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:46.659+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:47.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:47.443+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:47.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:47.695+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:48.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:48.479+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:48.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:48.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:49.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:49.455+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:49.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:49.767+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:50.421 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:50.415+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:50.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:50.803+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:51.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:51.439+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:51.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:51.819+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:52.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:52.403+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:52.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:52.839+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:53.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:53.387+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:53.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:53.807+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:54.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:54.347+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:54.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:54.791+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:55.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:55.367+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:55.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:55.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:56.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:56.415+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:56.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:56.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:57.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:57.411+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:57.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:57.771+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:58.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:58.459+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:58.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:58.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:59.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:03:59.471+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:03:59.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:03:59.783+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:00.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:00.519+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:00.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:00.767+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:01.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:01.479+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:01.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:01.735+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:02.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:02.522+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:02.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:02.742+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:03.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:03.562+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:03.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:03.790+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:04.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:04.582+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:04.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:04.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:05.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:05.610+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:05.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:05.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:06.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:06.658+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:06.813 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:06.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:07.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:07.702+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:07.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:07.786+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:08.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:08.726+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:08.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:08.826+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:09.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:09.722+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:09.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:09.874+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:10.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:10.758+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:10.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:10.850+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:11.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:11.738+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:11.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:11.846+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:12.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:12.706+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:12.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:12.802+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:13.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:13.754+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:13.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:13.774+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:14.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:14.786+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:14.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:14.822+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:15.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:15.738+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:15.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:15.782+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:16.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:16.782+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:16.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:16.822+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:17.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:17.790+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:17.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:17.822+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:18.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:18.762+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:18.873 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:18.866+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:19.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:19.782+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:19.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:19.886+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:20.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:20.786+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:20.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:20.898+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:21.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:21.742+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:21.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:21.870+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:22.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:22.782+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:22.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:22.862+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:23.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:23.750+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:23.878 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:23.870+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:24.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:24.786+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:24.902 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:24.898+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:25.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:25.778+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:25.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:25.906+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:26.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:26.786+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:26.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:26.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:27.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:27.738+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:27.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:27.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:28.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:28.742+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:29.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:28.994+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:29.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:29.758+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:30.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:30.046+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:30.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:30.754+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:31.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:31.090+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:31.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:31.802+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:32.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:32.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:32.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:32.850+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:33.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:33.018+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:33.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:33.822+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:34.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:34.046+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:34.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:34.858+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:35.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:35.034+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:35.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:35.870+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:36.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:36.002+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:36.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:36.894+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:36.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:36.982+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:37.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:37.898+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:37.964 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:37.958+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:38.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:38.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:38.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:38.942+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:39.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:39.902+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:39.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:39.902+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:40.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:40.886+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:40.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:40.914+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:41.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:41.870+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:41.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:41.934+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:42.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:42.910+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:42.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:42.982+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:43.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:43.942+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:44.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:44.018+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:44.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:44.926+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:45.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:45.030+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:45.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:45.946+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:46.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:46.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:46.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:46.926+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:47.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:47.094+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:47.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:47.878+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:48.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:48.126+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:48.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:48.866+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:49.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:49.102+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:49.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:49.846+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:50.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:50.074+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:50.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:50.842+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:51.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:51.114+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:51.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:51.814+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:52.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:52.154+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:52.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:52.846+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:53.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:53.198+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:53.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:53.830+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:54.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:54.234+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:54.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:54.878+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:55.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:55.226+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:55.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:55.834+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:56.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:56.218+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:56.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:56.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:57.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:57.258+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:57.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:57.794+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:58.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:58.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:58.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:58.825+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:59.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:04:59.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:04:59.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:04:59.805+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:00.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:00.329+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:00.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:00.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:01.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:01.297+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:01.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:01.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:02.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:02.269+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:02.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:02.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:03.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:03.237+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:03.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:03.753+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:04.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:04.209+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:04.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:04.781+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:05.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:05.209+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:05.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:05.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:06.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:06.225+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:06.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:06.769+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:07.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:07.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:07.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:07.777+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:08.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:08.213+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:08.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:08.769+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:09.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:09.189+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:09.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:09.721+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:10.196 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:10.189+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:10.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:10.729+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:11.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:11.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:11.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:11.757+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:12.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:12.133+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:12.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:12.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:13.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:13.181+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:13.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:13.761+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:14.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:14.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:14.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:14.729+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:15.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:15.165+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:15.694 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:15.685+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:16.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:16.213+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:16.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:16.693+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:17.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:17.245+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:17.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:17.741+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:18.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:18.249+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:18.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:18.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:19.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:19.205+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:19.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:19.801+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:20.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:20.237+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:20.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:20.853+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:21.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:21.225+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:21.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:21.821+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:22.196 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:22.189+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:22.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:22.801+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:23.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:23.225+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:23.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:23.765+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:24.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:24.201+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:24.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:24.745+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:25.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:25.205+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:25.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:25.749+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:26.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:26.165+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:26.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:26.705+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:27.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:27.189+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:27.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:27.705+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:28.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:28.221+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:28.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:28.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:29.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:29.233+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:29.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:29.697+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:30.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:30.205+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:30.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:30.681+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:31.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:31.221+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:31.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:31.697+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:32.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:32.225+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:32.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:32.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:33.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:33.221+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:33.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:33.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:34.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:34.253+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:34.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:34.705+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:35.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:35.245+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:35.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:35.693+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:36.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:36.269+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:36.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:36.705+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:37.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:37.297+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:37.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:37.753+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:38.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:38.293+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:38.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:38.717+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:39.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:39.261+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:39.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:39.757+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:40.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:40.305+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:40.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:40.757+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:41.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:41.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:41.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:41.717+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:42.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:42.253+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:42.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:42.709+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:43.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:43.237+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:43.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:43.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:44.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:44.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:44.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:44.753+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:45.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:45.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:45.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:45.789+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:46.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:46.213+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:46.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:46.841+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:47.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:47.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:47.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:47.877+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:48.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:48.309+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:48.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:48.925+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:49.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:49.305+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:49.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:49.941+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:50.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:50.273+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:50.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:50.921+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:51.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:51.253+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:51.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:51.933+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:52.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:52.285+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:52.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:52.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:53.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:53.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:53.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:53.921+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:54.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:54.281+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:54.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:54.917+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:55.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:55.305+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:55.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:55.873+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:56.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:56.309+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:56.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:56.901+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:57.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:57.309+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:57.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:57.929+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:58.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:58.309+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:58.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:58.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:59.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:05:59.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:05:59.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:05:59.933+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:00.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:00.337+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:00.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:00.965+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:01.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:01.297+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:02.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:02.001+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:02.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:02.253+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:02.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:02.973+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:03.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:03.273+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:03.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:03.937+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:04.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:04.233+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:04.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:04.913+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:05.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:05.201+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:05.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:05.877+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:06.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:06.233+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:06.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:06.877+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:07.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:07.245+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:07.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:07.909+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:08.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:08.281+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:08.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:08.941+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:09.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:09.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:09.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:09.925+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:10.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:10.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:10.978 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:10.973+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:11.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:11.317+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:11.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:11.929+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:12.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:12.357+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:12.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:12.889+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:13.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:13.385+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:13.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:13.877+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:14.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:14.437+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:14.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:14.909+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:15.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:15.389+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:15.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:15.861+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:16.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:16.381+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:16.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:16.837+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:17.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:17.392+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:17.807 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:17.800+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:18.429 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:18.420+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:18.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:18.780+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:19.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:19.440+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:19.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:19.820+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:20.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:20.464+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:20.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:20.820+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:21.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:21.440+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:21.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:21.780+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:22.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:22.408+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:22.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:22.756+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:23.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:23.432+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:23.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:23.784+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:24.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:24.440+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:24.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:24.780+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:25.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:25.452+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:25.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:25.772+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:26.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:26.484+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:26.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:26.780+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:27.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:27.476+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:27.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:27.808+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:28.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:28.456+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:28.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:28.772+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:29.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:29.416+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:29.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:29.724+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:30.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:30.456+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:30.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:30.700+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:31.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:31.464+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:31.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:31.656+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:32.455 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:32.448+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:32.701 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:32.692+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:33.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:33.400+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:33.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:33.688+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:34.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:34.428+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:34.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:34.680+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:35.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:35.392+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:35.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:35.648+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:36.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:36.380+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:36.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:36.612+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:37.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:37.400+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:37.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:37.584+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:38.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:38.440+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:38.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:38.568+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:39.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:39.428+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:39.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:39.556+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:40.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:40.432+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:40.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:40.588+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:41.476 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:41.468+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:41.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:41.576+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:42.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:42.452+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:42.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:42.588+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:43.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:43.464+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:43.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:43.584+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:44.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:44.496+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:44.631 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:44.624+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:45.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:45.456+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:45.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:45.608+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:46.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:46.460+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:46.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:46.652+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:47.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:47.440+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:47.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:47.692+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:48.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:48.460+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:48.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:48.648+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:49.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:49.444+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:49.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:49.632+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:50.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:50.448+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:50.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:50.588+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:51.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:51.412+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:51.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:51.584+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:52.386 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:52.380+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:52.578 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:52.572+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:53.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:53.388+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:53.571 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:53.564+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:54.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:54.356+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:54.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:54.616+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:55.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:55.320+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:55.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:55.640+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:56.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:56.308+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:56.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:56.688+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:57.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:57.308+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:57.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:57.692+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:58.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:58.308+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:58.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:58.672+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:59.330 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:06:59.324+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:06:59.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:06:59.644+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:00.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:00.372+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:00.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:00.644+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:01.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:01.360+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:01.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:01.616+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:02.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:02.336+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:02.615 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:02.608+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:03.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:03.296+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:03.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:03.624+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:04.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:04.344+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:04.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:04.636+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:05.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:05.304+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:05.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:05.656+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:06.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:06.260+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:06.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:06.612+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:07.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:07.312+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:07.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:07.656+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:08.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:08.272+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:08.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:08.680+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:09.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:09.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:09.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:09.720+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:10.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:10.240+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:10.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:10.744+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:11.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:11.272+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:11.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:11.776+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:12.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:12.244+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:12.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:12.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:13.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:13.232+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:13.807 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:13.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:14.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:14.228+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:14.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:14.836+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:15.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:15.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:15.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:15.852+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:16.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:16.284+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:16.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:16.880+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:17.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:17.312+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:17.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:17.928+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:18.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:18.312+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:18.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:18.964+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:19.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:19.296+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:19.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:19.928+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:20.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:20.328+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:20.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:20.920+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:21.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:21.352+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:21.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:21.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:22.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:22.312+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:22.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:22.888+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:23.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:23.344+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:23.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:23.892+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:24.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:24.376+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:24.943 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:24.940+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:25.413 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:25.408+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:25.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:25.952+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:26.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:26.400+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:26.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:26.912+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:27.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:27.404+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:27.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:27.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:28.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:28.360+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:28.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:28.967+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:29.413 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:29.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:29.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:29.923+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:30.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:30.395+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:30.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:30.955+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:31.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:31.443+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:31.966 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:31.963+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:32.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:32.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:32.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:32.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:33.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:33.391+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:33.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:33.943+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:34.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:34.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:34.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:34.951+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:35.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:35.347+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:35.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:35.907+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:36.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:36.387+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:36.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:36.867+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:37.380 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:37.375+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:37.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:37.875+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:38.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:38.387+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:38.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:38.847+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:39.413 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:39.411+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:39.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:39.827+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:40.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:40.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:40.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:40.819+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:41.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:41.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:41.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:41.811+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:42.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:42.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:42.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:42.823+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:43.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:43.387+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:43.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:43.827+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:44.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:44.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:44.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:44.803+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:45.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:45.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:45.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:45.771+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:46.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:46.359+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:46.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:46.723+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:47.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:47.383+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:47.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:47.767+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:48.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:48.359+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:48.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:48.803+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:49.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:49.315+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:49.807 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:49.803+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:50.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:50.351+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:50.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:50.819+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:51.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:51.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:51.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:51.863+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:52.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:52.391+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:52.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:52.903+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:53.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:53.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:53.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:53.891+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:54.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:54.343+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:54.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:54.939+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:55.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:55.359+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:55.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:55.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:56.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:56.339+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:56.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:56.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:57.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:57.335+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:57.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:57.947+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:58.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:58.375+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:58.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:07:58.979+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:07:59.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:07:59.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:00.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:00.015+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:00.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:00.411+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:01.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:01.019+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:01.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:01.391+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:01.997 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:01.995+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:02.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:02.359+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:03.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:03.027+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:03.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:03.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:04.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:04.003+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:04.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:04.447+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:05.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:04.999+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:05.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:05.443+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:05.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:05.959+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:06.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:06.471+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:06.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:06.931+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:07.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:07.439+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:07.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:07.919+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:08.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:08.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:08.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:08.927+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:09.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:09.391+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:09.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:09.927+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:10.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:10.391+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:10.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:10.939+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:11.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:11.439+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:11.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:11.907+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:12.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:12.455+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:12.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:12.931+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:13.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:13.495+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:13.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:13.971+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:14.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:14.523+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:15.017 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:15.015+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:15.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:15.515+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:16.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:16.035+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:16.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:16.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:17.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:17.067+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:17.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:17.511+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:18.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:18.027+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:18.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:18.543+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:19.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:19.059+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:19.524 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:19.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:20.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:20.043+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:20.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:20.515+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:21.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:21.043+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:21.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:21.471+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:22.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:21.999+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:22.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:22.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:23.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:23.007+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:23.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:23.511+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:23.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:23.987+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:24.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:24.531+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:24.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:24.951+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:25.587 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:25.583+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:25.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:25.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:26.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:26.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:26.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:26.971+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:27.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:27.603+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:27.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:27.979+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:28.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:28.655+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:28.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:28.991+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:29.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:29.651+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:30.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:30.007+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:30.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:30.695+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:30.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:30.971+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:31.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:31.651+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:31.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:31.955+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:32.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:32.619+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:32.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:32.907+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:33.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:33.667+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:33.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:33.943+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:34.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:34.707+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:34.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:34.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:35.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:35.711+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:35.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:35.931+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:36.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:36.755+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:36.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:36.915+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:37.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:37.763+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:37.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:37.875+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:38.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:38.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:38.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:38.891+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:39.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:39.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:39.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:39.851+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:40.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:40.751+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:40.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:40.835+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:41.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:41.743+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:41.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:41.855+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:42.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:42.723+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:42.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:42.843+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:43.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:43.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:43.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:43.822+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:44.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:44.774+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:44.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:44.822+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:45.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:45.790+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:45.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:45.870+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:46.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:46.762+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:46.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:46.834+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:47.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:47.738+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:47.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:47.806+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:48.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:48.762+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:48.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:48.806+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:49.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:49.714+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:49.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:49.814+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:50.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:50.714+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:50.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:50.782+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:51.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:51.758+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:51.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:51.822+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:52.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:52.786+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:52.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:52.790+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:53.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:53.746+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:53.825 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:53.822+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:54.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:54.718+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:54.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:54.834+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:55.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:55.746+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:55.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:55.870+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:56.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:56.790+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:56.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:56.862+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:57.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:57.834+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:57.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:57.834+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:58.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:58.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:58.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:58.830+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:59.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:08:59.782+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:08:59.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:08:59.790+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:00.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:00.778+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:00.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:00.822+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:01.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:01.758+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:01.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:01.846+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:02.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:02.766+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:02.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:02.874+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:03.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:03.746+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:03.893 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:03.890+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:04.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:04.702+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:04.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:04.886+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:05.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:05.690+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:05.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:05.910+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:06.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:06.670+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:06.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:06.862+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:07.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:07.642+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:07.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:07.906+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:08.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:08.674+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:08.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:08.906+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:09.667 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:09.662+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:09.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:09.934+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:10.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:10.674+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:10.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:10.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:11.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:11.666+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:11.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:11.950+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:12.671 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:12.666+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:12.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:12.962+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:13.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:13.654+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:13.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:13.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:14.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:14.698+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:15.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:15.010+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:15.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:15.734+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:16.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:16.002+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:16.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:16.782+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:17.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:17.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:17.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:17.830+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:18.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:18.018+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:18.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:18.858+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:18.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:18.990+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:19.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:19.810+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:20.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:20.002+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:20.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:20.830+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:21.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:21.034+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:21.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:21.818+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:22.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:22.046+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:22.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:22.810+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:23.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:23.050+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:23.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:23.770+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:24.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:24.014+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:24.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:24.726+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:25.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:25.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:25.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:25.694+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:26.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:26.014+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:26.691 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:26.686+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:27.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:27.002+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:27.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:27.706+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:27.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:27.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:28.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:28.722+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:28.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:28.982+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:29.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:29.730+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:29.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:29.966+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:30.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:30.714+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:30.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:30.938+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:31.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:31.746+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:31.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:31.910+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:32.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:32.770+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:32.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:32.886+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:33.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:33.810+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:33.914 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:33.910+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:34.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:34.794+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:34.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:34.954+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:35.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:35.774+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:35.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:35.906+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:36.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:36.754+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:36.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:36.906+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:37.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:37.802+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:37.926 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:37.922+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:38.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:38.766+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:38.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:38.938+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:39.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:39.734+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:39.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:39.946+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:40.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:40.686+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:40.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:40.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:41.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:41.654+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:41.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:41.962+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:42.640 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:42.634+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:42.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:42.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:43.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:43.618+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:44.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:44.018+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:44.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:44.618+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:45.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:45.002+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:45.602 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:45.598+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:46.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:46.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:46.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:46.614+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:47.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:47.022+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:47.618 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:47.614+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:48.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:48.058+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:48.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:48.582+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:49.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:49.058+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:49.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:49.545+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:50.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:50.101+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:50.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:50.581+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:51.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:51.077+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:51.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:51.561+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:52.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:52.125+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:52.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:52.545+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:53.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:53.161+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:53.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:53.545+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:54.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:54.173+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:54.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:54.545+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:55.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:55.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:55.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:55.581+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:56.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:56.201+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:56.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:56.577+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:57.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:57.181+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:57.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:57.561+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:58.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:58.169+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:58.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:58.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:59.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:09:59.181+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:09:59.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:09:59.561+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:00.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:00.221+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:00.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:00.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:01.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:01.193+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:01.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:01.489+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:02.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:02.157+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:02.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:02.485+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:03.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:03.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:03.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:03.453+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:04.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:04.169+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:04.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:04.445+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:05.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:05.201+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:05.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:05.461+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:06.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:06.165+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:06.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:06.485+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:07.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:07.149+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:07.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:07.505+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:08.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:08.141+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:08.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:08.513+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:09.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:09.145+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:09.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:09.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:10.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:10.173+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:10.502 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:10.497+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:11.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:11.197+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:11.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:11.533+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:12.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:12.205+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:12.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:12.577+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:13.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:13.185+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:13.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:13.581+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:14.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:14.221+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:14.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:14.605+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:15.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:15.225+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:15.631 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:15.625+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:16.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:16.249+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:16.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:16.669+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:17.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:17.217+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:17.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:17.677+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:18.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:18.213+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:18.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:18.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:19.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:19.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:19.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:19.705+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:20.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:20.277+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:20.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:20.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:21.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:21.309+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:21.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:21.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:22.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:22.265+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:22.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:22.737+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:23.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:23.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:23.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:23.785+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:24.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:24.233+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:24.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:24.785+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:25.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:25.269+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:25.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:25.821+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:26.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:26.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:26.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:26.841+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:27.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:27.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:27.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:27.793+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:28.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:28.285+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:28.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:28.797+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:29.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:29.337+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:29.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:29.781+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:30.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:30.333+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:30.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:30.737+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:31.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:31.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:31.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:31.709+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:32.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:32.317+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:32.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:32.685+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:33.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:33.293+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:33.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:33.637+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:34.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:34.269+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:34.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:34.609+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:35.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:35.265+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:35.633 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:35.629+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:36.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:36.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:36.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:36.673+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:37.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:37.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:37.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:37.633+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:38.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:38.269+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:38.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:38.669+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:39.326 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:39.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:39.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:39.717+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:40.335 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:40.329+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:40.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:40.741+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:41.380 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:41.377+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:41.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:41.789+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:42.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:42.421+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:42.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:42.741+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:43.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:43.409+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:43.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:43.741+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:44.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:44.417+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:44.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:44.765+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:45.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:45.377+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:45.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:45.781+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:46.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:46.361+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:46.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:46.781+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:47.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:47.337+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:47.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:47.769+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:48.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:48.325+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:48.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:48.805+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:49.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:49.337+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:49.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:49.841+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:50.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:50.385+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:50.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:50.816+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:51.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:51.396+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:51.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:51.768+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:52.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:52.404+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:52.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:52.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:53.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:53.420+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:53.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:53.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:54.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:54.448+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:54.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:54.784+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:55.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:55.456+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:55.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:55.768+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:56.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:56.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:56.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:56.752+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:57.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:57.516+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:57.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:57.752+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:58.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:58.568+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:58.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:58.756+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:59.619 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:10:59.612+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:10:59.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:10:59.772+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:00.623 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:00.616+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:00.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:00.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:01.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:01.652+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:01.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:01.716+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:02.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:02.664+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:02.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:02.684+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:03.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:03.648+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:03.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:03.660+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:04.660 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:04.652+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:04.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:04.660+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:05.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:05.668+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:05.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:05.680+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:06.669 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:06.664+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:06.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:06.704+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:07.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:07.688+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:07.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:07.720+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:08.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:08.700+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:08.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:08.748+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:09.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:09.744+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:09.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:09.752+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:10.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:10.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:10.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:10.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:11.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:11.752+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:11.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:11.812+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:12.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:12.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:12.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:12.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:13.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:13.740+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:13.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:13.840+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:14.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:14.776+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:14.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:14.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:15.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:15.776+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:15.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:15.844+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:16.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:16.732+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:16.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:16.880+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:17.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:17.732+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:17.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:17.864+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:18.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:18.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:18.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:18.864+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:19.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:19.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:19.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:19.828+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:20.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:20.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:20.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:20.804+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:21.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:21.768+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:21.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:21.816+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:22.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:22.724+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:22.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:22.772+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:23.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:23.748+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:23.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:23.752+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:24.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:24.700+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:24.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:24.780+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:25.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:25.676+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:25.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:25.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:26.718 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:26.712+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:26.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:26.796+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:27.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:27.752+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:27.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:27.768+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:28.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:28.708+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:28.752 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:28.744+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:29.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:29.696+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:29.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:29.744+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:30.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:30.668+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:30.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:30.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:31.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:31.688+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:31.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:31.768+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:32.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:32.740+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:32.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:32.800+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:33.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:33.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:33.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:33.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:34.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:34.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:34.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:34.832+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:35.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:35.748+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:35.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:35.812+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:36.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:36.700+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:36.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:36.856+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:37.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:37.684+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:37.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:37.880+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:38.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:38.652+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:38.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:38.924+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:39.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:39.676+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:39.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:39.928+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:40.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:40.704+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:40.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:40.920+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:41.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:41.680+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:41.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:41.884+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:42.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:42.708+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:42.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:42.888+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:43.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:43.708+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:43.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:43.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:44.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:44.660+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:44.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:44.876+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:45.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:45.648+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:45.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:45.844+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:46.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:46.688+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:46.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:46.860+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:47.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:47.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:47.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:47.832+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:48.668 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:48.667+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:48.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:48.859+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:49.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:49.675+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:49.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:49.831+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:50.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:50.699+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:50.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:50.883+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:51.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:51.671+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:51.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:51.923+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:52.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:52.715+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:52.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:52.899+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:53.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:53.723+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:53.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:53.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:54.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:54.695+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:54.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:54.955+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:55.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:55.735+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:55.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:55.959+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:56.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:56.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:56.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:56.955+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:57.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:57.807+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:57.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:57.919+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:58.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:58.815+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:58.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:58.923+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:59.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:11:59.815+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:11:59.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:11:59.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:00.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:00.791+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:00.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:00.923+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:01.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:01.743+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:01.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:01.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:02.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:02.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:02.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:02.875+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:03.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:03.807+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:03.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:03.879+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:04.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:04.823+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:04.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:04.923+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:05.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:05.843+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:05.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:05.875+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:06.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:06.835+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:06.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:06.875+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:07.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:07.843+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:07.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:07.883+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:08.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:08.843+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:08.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:08.911+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:09.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:09.819+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:09.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:09.891+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:10.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:10.847+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:10.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:10.883+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:11.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:11.811+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:11.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:11.859+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:12.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:12.843+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:12.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:12.871+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:13.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:13.887+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:13.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:13.923+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:14.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:14.935+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:14.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:14.939+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:15.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:15.931+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:15.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:15.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:16.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:16.903+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:16.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:16.907+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:17.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:17.911+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:17.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:17.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:18.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:18.919+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:18.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:18.963+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:19.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:19.907+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:19.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:19.995+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:20.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:20.871+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:20.963 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:20.959+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:21.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:21.883+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:21.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:21.947+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:22.908 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:22.907+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:22.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:22.919+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:23.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:23.887+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:23.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:23.967+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:24.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:24.851+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:24.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:24.983+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:25.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:25.831+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:26.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:26.027+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:26.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:26.839+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:27.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:27.015+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:27.854 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:27.851+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:28.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:28.031+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:28.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:28.903+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:29.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:29.079+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:29.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:29.875+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:30.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:30.067+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:30.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:30.827+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:31.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:31.111+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:31.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:31.843+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:32.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:32.111+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:32.895 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:32.891+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:33.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:33.071+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:33.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:33.895+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:34.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:34.111+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:34.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:34.899+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:35.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:35.091+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:35.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:35.951+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:36.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:36.099+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:36.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:36.915+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:37.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:37.111+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:37.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:37.899+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:38.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:38.115+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:38.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:38.899+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:39.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:39.071+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:39.854 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:39.851+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:40.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:40.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:40.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:40.887+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:41.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:41.083+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:41.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:41.851+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:42.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:42.123+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:42.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:42.903+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:43.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:43.091+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:43.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:43.927+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:44.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:44.083+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:44.963 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:44.958+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:45.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:45.114+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:46.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:45.994+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:46.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:46.082+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:47.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:47.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:47.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:47.062+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:48.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:48.034+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:48.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:48.082+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:49.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:49.046+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:49.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:49.086+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:50.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:50.010+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:50.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:50.126+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:50.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:50.970+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:51.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:51.162+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:51.978 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:51.974+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:52.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:52.162+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:52.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:52.974+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:53.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:53.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:53.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:53.986+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:54.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:54.106+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:54.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:54.978+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:55.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:55.082+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:55.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:55.990+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:56.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:56.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:57.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:57.018+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:57.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:57.042+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:58.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:58.006+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:58.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:58.026+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:59.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:12:59.006+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:12:59.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:12:59.058+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:00.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:00.030+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:00.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:00.078+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:01.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:01.058+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:01.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:01.122+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:02.017 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:02.014+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:02.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:02.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:03.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:03.018+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:03.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:03.174+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:04.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:04.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:04.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:04.222+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:05.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:05.010+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:05.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:05.202+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:06.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:06.050+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:06.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:06.182+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:07.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:07.046+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:07.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:07.182+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:08.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:08.082+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:08.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:08.174+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:09.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:09.062+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:09.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:09.158+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:10.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:10.022+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:10.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:10.114+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:11.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:11.006+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:11.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:11.114+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:11.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:11.994+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:12.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:12.102+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:13.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:12.998+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:13.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:13.070+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:13.982 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:13.978+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:14.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:14.098+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:15.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:15.022+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:15.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:15.126+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:16.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:16.010+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:16.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:16.126+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:17.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:17.030+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:17.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:17.118+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:18.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:18.022+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:18.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:18.102+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:18.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:18.974+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:19.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:19.082+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:19.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:19.946+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:20.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:20.046+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:20.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:20.966+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:21.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:21.098+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:21.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:21.982+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:22.083 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:22.078+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:23.010 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:23.006+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:23.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:23.070+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:24.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:24.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:24.070 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:24.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:24.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:24.990+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:25.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:25.058+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:26.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:26.022+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:26.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:26.090+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:27.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:26.998+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:27.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:27.110+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:27.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:27.974+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:28.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:28.078+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:28.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:28.938+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:29.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:29.102+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:29.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:29.934+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:30.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:30.154+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:30.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:30.978+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:31.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:31.170+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:32.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:32.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:32.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:32.222+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:33.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:33.002+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:33.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:33.174+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:34.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:34.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:34.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:34.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:35.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:35.074+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:35.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:35.158+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:36.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:36.098+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:36.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:36.126+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:37.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:37.062+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:37.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:37.094+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:38.041 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:38.038+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:38.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:38.094+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:38.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:38.989+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:39.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:39.097+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:39.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:39.969+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:40.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:40.109+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:41.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:41.009+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:41.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:41.081+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:41.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:41.989+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:42.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:42.129+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:43.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:42.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:43.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:43.097+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:44.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:44.021+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:44.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:44.109+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:45.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:45.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:45.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:45.081+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:46.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:46.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:46.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:46.109+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:47.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:47.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:47.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:47.077+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:48.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:48.045+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:48.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:48.093+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:49.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:49.069+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:49.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:49.081+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:50.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:50.077+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:50.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:50.093+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:51.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:51.041+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:51.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:51.125+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:52.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:52.077+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:52.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:52.125+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:53.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:53.037+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:53.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:53.117+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:54.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:54.069+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:54.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:54.113+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:55.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:55.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:55.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:55.065+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:56.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:56.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:56.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:56.093+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:57.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:57.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:57.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:57.093+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:58.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:58.037+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:58.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:58.129+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:59.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:59.041+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:13:59.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:13:59.133+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:00.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:13:59.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:00.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:00.109+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:01.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:00.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:01.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:01.137+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:02.017 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:02.013+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:02.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:02.145+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:03.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:03.029+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:03.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:03.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:04.084 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:04.081+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:04.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:04.133+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:05.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:05.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:05.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:05.133+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:06.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:06.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:06.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:06.109+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:07.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:07.041+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:07.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:07.149+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:08.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:08.045+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:08.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:08.201+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:09.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:09.085+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:09.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:09.153+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:10.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:10.057+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:10.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:10.121+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:11.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:11.045+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:11.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:11.145+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:12.023 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:12.017+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:12.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:12.153+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:13.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:12.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:13.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:13.161+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:13.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:13.957+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:14.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:14.173+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:14.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:14.969+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:15.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:15.153+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:15.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:15.933+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:16.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:16.169+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:16.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:16.977+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:17.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:17.181+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:18.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:18.005+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:18.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:18.217+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:18.966 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:18.961+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:19.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:19.185+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:19.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:19.925+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:20.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:20.153+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:20.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:20.961+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:21.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:21.113+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:21.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:21.969+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:22.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:22.145+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:22.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:22.957+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:23.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:23.129+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:23.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:23.949+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:24.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:24.085+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:24.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:24.977+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:25.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:25.073+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:25.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:25.973+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:26.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:26.033+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:26.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:26.993+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:27.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:27.005+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:27.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:27.965+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:28.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:27.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:28.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:28.989+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:29.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:29.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:30.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:30.009+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:30.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:30.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:30.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:30.981+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:31.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:31.029+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:31.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:31.972+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:32.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:32.044+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:32.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:32.988+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:33.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:33.076+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:33.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:33.992+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:34.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:34.080+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:35.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:35.036+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:35.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:35.044+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:36.010 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:36.004+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:36.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:36.056+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:36.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:36.992+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:37.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:37.056+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:38.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:38.008+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:38.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:38.068+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:39.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:39.004+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:39.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:39.088+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:40.023 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:40.016+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:40.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:40.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:41.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:41.040+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:41.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:41.164+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:42.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:42.024+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:42.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:42.196+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:43.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:43.016+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:43.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:43.152+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:43.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:43.980+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:44.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:44.108+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:44.997 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:44.992+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:45.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:45.144+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:46.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:46.020+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:46.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:46.192+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:47.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:47.036+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:47.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:47.148+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:48.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:48.044+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:48.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:48.188+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:49.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:49.076+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:49.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:49.236+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:50.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:50.072+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:50.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:50.272+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:51.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:51.100+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:51.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:51.236+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:52.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:52.104+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:52.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:52.288+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:53.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:53.124+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:53.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:53.272+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:54.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:54.160+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:54.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:54.248+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:55.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:55.176+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:55.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:55.284+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:56.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:56.160+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:56.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:56.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:57.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:57.188+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:57.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:57.256+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:58.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:58.160+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:58.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:58.260+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:59.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:14:59.208+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:14:59.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:14:59.284+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:00.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:00.236+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:00.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:00.240+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:01.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:01.232+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:01.274 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:01.268+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:02.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:02.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:02.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:02.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:03.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:03.284+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:03.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:03.288+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:04.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:04.244+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:04.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:04.244+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:05.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:05.220+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:05.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:05.284+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:06.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:06.204+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:06.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:06.332+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:07.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:07.244+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:07.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:07.360+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:08.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:08.224+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:08.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:08.372+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:09.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:09.200+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:09.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:09.376+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:10.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:10.172+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:10.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:10.388+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:11.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:11.208+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:11.421 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:11.420+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:12.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:12.220+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:12.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:12.388+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:13.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:13.172+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:13.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:13.432+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:14.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:14.136+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:14.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:14.388+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:15.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:15.120+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:15.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:15.368+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:16.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:16.120+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:16.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:16.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:17.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:17.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:17.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:17.360+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:18.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:18.080+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:18.340 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:18.340+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:19.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:19.100+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:19.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:19.372+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:20.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:20.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:20.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:20.420+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:21.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:21.172+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:21.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:21.460+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:22.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:22.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:22.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:22.448+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:23.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:23.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:23.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:23.423+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:24.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:24.135+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:24.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:24.451+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:25.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:25.171+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:25.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:25.495+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:26.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:26.199+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:26.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:26.471+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:27.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:27.195+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:27.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:27.515+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:28.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:28.223+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:28.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:28.519+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:29.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:29.227+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:29.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:29.507+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:30.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:30.219+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:30.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:30.543+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:31.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:31.267+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:31.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:31.547+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:32.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:32.251+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:32.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:32.563+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:33.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:33.227+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:33.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:33.531+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:34.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:34.275+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:34.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:34.523+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:35.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:35.239+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:35.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:35.535+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:36.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:36.251+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:36.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:36.515+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:37.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:37.295+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:37.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:37.503+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:38.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:38.315+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:38.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:38.475+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:39.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:39.347+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:39.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:39.439+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:40.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:40.367+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:40.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:40.455+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:41.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:41.343+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:41.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:41.487+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:42.351 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:42.347+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:42.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:42.531+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:43.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:43.379+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:43.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:43.559+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:44.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:44.351+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:44.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:44.575+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:45.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:45.359+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:45.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:45.579+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:46.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:46.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:46.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:46.547+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:47.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:47.395+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:47.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:47.555+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:48.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:48.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:48.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:48.531+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:49.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:49.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:49.538 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:49.535+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:50.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:50.435+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:50.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:50.567+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:51.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:51.443+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:51.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:51.547+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:52.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:52.491+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:52.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:52.499+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:53.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:53.463+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:53.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:53.511+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:54.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:54.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:54.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:54.531+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:55.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:55.455+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:55.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:55.483+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:56.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:56.475+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:56.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:56.495+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:57.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:57.491+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:57.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:57.503+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:58.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:58.479+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:58.504 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:58.499+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:59.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:15:59.503+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:15:59.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:15:59.515+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:00.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:00.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:00.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:00.551+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:01.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:01.499+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:01.595 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:01.591+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:02.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:02.531+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:02.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:02.555+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:03.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:03.571+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:03.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:03.579+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:04.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:04.543+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:04.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:04.619+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:05.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:05.543+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:05.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:05.651+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:06.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:06.591+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:06.655 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:06.651+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:07.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:07.631+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:07.687 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:07.683+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:08.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:08.631+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:08.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:08.643+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:09.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:09.635+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:09.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:09.635+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:10.613 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:10.611+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:10.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:10.635+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:11.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:11.623+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:11.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:11.639+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:12.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:12.603+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:12.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:12.647+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:13.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:13.627+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:13.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:13.627+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:14.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:14.591+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:14.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:14.651+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:15.641 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:15.639+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:15.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:15.687+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:16.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:16.610+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:16.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:16.650+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:17.627 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:17.622+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:17.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:17.678+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:18.655 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:18.650+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:18.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:18.718+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:19.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:19.650+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:19.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:19.742+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:20.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:20.626+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:20.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:20.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:21.633 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:21.630+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:21.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:21.746+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:22.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:22.610+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:22.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:22.742+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:23.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:23.598+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:23.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:23.734+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:24.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:24.594+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:24.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:24.750+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:25.634 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:25.630+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:25.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:25.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:26.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:26.590+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:26.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:26.786+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:27.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:27.618+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:27.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:27.802+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:28.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:28.646+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:28.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:28.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:29.635 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:29.630+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:29.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:29.782+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:30.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:30.602+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:30.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:30.782+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:31.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:31.610+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:31.803 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:31.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:32.636 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:32.630+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:32.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:32.810+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:33.602 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:33.598+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:33.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:33.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:34.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:34.562+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:34.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:34.738+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:35.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:35.562+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:35.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:35.714+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:36.615 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:36.610+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:36.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:36.726+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:37.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:37.566+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:37.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:37.738+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:38.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:38.518+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:38.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:38.750+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:39.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:39.506+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:39.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:39.794+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:40.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:40.502+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:40.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:40.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:41.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:41.526+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:41.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:41.794+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:42.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:42.482+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:42.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:42.782+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:43.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:43.438+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:43.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:43.822+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:44.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:44.430+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:44.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:44.870+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:45.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:45.446+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:45.887 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:45.882+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:46.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:46.450+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:46.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:46.850+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:47.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:47.430+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:47.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:47.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:48.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:48.478+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:48.803 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:48.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:49.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:49.490+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:49.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:49.794+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:50.538 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:50.534+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:50.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:50.814+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:51.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:51.526+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:51.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:51.790+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:52.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:52.502+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:52.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:52.774+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:53.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:53.458+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:53.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:53.774+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:54.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:54.502+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:54.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:54.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:55.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:55.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:55.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:55.770+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:56.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:56.514+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:56.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:56.730+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:57.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:57.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:57.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:57.714+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:58.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:58.498+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:58.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:58.734+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:59.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:16:59.502+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:16:59.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:16:59.730+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:00.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:00.506+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:00.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:00.738+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:01.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:01.554+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:01.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:01.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:02.571 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:02.566+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:02.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:02.746+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:03.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:03.534+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:03.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:03.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:04.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:04.514+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:04.820 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:04.818+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:05.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:05.498+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:05.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:05.782+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:06.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:06.498+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:06.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:06.818+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:07.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:07.538+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:07.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:07.790+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:08.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:08.494+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:08.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:08.814+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:09.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:09.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:09.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:09.814+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:10.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:10.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:10.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:10.825+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:11.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:11.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:11.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:11.877+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:12.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:12.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:12.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:12.833+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:13.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:13.545+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:13.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:13.849+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:14.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:14.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:14.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:14.885+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:15.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:15.481+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:15.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:15.853+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:16.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:16.481+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:16.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:16.889+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:17.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:17.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:17.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:17.909+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:18.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:18.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:18.952 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:18.945+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:19.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:19.553+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:19.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:19.977+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:20.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:20.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:20.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:20.969+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:21.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:21.493+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:21.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:21.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:22.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:22.453+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:22.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:22.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:23.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:23.457+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:23.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:23.973+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:24.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:24.413+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:25.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:25.009+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:25.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:25.445+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:26.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:26.009+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:26.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:26.409+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:27.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:27.029+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:27.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:27.449+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:28.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:28.069+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:28.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:28.441+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:29.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:29.117+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:29.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:29.449+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:30.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:30.161+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:30.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:30.433+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:31.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:31.197+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:31.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:31.385+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:32.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:32.217+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:32.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:32.377+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:33.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:33.241+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:33.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:33.413+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:34.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:34.221+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:34.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:34.393+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:35.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:35.261+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:35.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:35.433+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:36.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:36.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:36.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:36.465+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:37.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:37.281+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:37.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:37.425+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:38.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:38.297+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:38.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:38.413+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:39.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:39.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:39.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:39.369+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:40.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:40.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:40.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:40.345+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:41.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:41.269+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:41.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:41.385+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:42.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:42.241+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:42.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:42.433+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:43.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:43.201+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:43.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:43.453+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:44.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:44.193+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:44.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:44.477+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:45.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:45.193+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:45.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:45.497+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:46.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:46.241+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:46.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:46.453+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:47.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:47.233+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:47.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:47.461+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:48.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:48.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:48.516 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:48.513+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:49.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:49.277+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:49.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:49.505+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:50.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:50.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:50.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:50.509+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:51.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:51.305+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:51.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:51.549+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:52.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:52.277+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:52.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:52.505+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:53.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:53.261+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:53.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:53.485+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:54.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:54.217+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:54.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:54.509+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:55.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:55.193+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:55.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:55.497+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:56.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:56.225+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:56.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:56.517+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:57.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:57.245+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:57.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:57.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:58.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:58.225+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:58.502 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:58.497+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:59.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:17:59.193+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:17:59.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:17:59.469+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:00.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:00.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:00.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:00.509+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:01.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:01.181+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:01.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:01.469+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:02.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:02.153+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:02.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:02.513+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:03.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:03.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:03.541 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:03.536+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:04.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:04.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:04.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:04.524+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:05.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:05.144+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:05.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:05.556+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:06.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:06.112+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:06.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:06.516+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:07.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:07.112+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:07.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:07.560+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:08.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:08.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:08.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:08.524+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:09.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:09.104+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:09.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:09.532+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:10.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:10.072+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:10.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:10.580+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:11.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:11.096+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:11.615 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:11.608+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:12.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:12.076+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:12.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:12.588+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:13.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:13.032+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:13.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:13.544+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:14.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:14.024+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:14.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:14.592+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:15.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:15.004+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:15.603 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:15.596+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:16.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:16.036+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:16.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:16.576+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:17.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:17.004+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:17.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:17.604+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:17.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:17.984+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:18.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:18.568+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:19.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:19.008+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:19.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:19.556+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:20.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:20.000+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:20.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:20.584+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:20.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:20.992+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:21.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:21.592+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:21.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:21.952+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:22.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:22.548+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:22.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:22.976+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:23.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:23.584+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:23.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:23.928+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:24.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:24.592+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:24.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:24.884+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:25.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:25.548+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:25.941 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:25.936+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:26.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:26.504+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:26.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:26.912+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:27.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:27.480+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:27.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:27.924+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:28.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:28.448+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:28.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:28.952+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:29.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:29.432+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:29.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:29.924+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:30.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:30.428+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:30.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:30.904+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:31.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:31.428+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:31.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:31.888+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:32.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:32.404+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:32.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:32.908+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:33.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:33.452+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:33.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:33.884+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:34.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:34.492+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:34.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:34.856+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:35.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:35.488+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:35.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:35.892+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:36.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:36.532+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:36.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:36.940+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:37.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:37.488+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:37.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:37.988+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:38.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:38.492+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:38.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:38.964+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:39.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:39.484+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:40.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:40.000+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:40.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:40.532+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:40.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:40.968+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:41.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:41.544+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:42.018 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:42.016+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:42.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:42.528+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:42.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:42.972+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:43.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:43.516+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:43.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:43.976+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:44.502 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:44.500+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:45.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:45.020+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:45.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:45.512+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:46.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:46.052+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:46.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:46.528+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:47.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:47.092+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:47.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:47.560+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:48.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:48.076+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:48.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:48.512+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:49.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:49.072+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:49.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:49.564+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:50.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:50.072+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:50.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:50.568+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:51.070 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:51.068+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:51.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:51.520+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:52.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:52.056+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:52.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:52.492+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:53.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:53.048+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:53.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:53.532+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:54.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:54.084+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:54.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:54.524+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:55.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:55.068+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:55.485 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:55.483+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:56.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:56.067+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:56.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:56.459+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:57.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:57.111+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:57.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:57.475+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:58.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:58.083+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:58.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:58.467+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:59.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:18:59.035+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:18:59.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:18:59.447+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:00.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:00.015+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:00.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:00.475+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:00.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:00.987+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:01.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:01.431+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:01.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:01.971+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:02.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:02.471+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:02.941 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:02.939+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:03.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:03.447+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:03.933 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:03.931+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:04.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:04.435+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:04.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:04.955+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:05.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:05.427+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:06.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:06.003+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:06.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:06.419+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:06.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:06.971+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:07.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:07.463+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:07.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:07.979+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:08.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:08.475+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:08.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:08.995+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:09.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:09.435+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:09.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:09.947+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:10.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:10.391+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:10.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:10.967+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:11.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:11.387+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:11.949 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:11.947+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:12.373 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:12.371+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:12.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:12.939+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:13.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:13.323+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:13.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:13.979+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:14.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:14.279+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:14.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:14.955+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:15.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:15.295+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:15.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:15.967+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:16.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:16.251+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:16.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:16.931+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:17.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:17.247+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:17.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:17.943+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:18.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:18.227+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:18.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:18.975+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:19.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:19.251+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:20.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:20.003+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:20.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:20.219+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:21.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:21.031+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:21.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:21.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:22.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:22.019+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:22.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:22.191+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:23.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:23.003+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:23.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:23.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:24.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:24.035+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:24.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:24.219+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:24.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:24.987+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:25.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:25.243+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:26.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:26.007+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:26.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:26.287+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:27.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:27.023+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:27.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:27.271+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:28.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:27.999+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:28.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:28.323+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:29.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:29.007+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:29.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:29.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:30.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:30.027+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:30.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:30.331+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:31.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:31.027+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:31.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:31.371+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:32.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:32.051+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:32.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:32.347+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:33.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:33.071+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:33.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:33.371+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:34.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:34.063+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:34.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:34.363+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:35.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:35.043+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:35.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:35.339+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:36.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:36.059+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:36.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:36.315+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:37.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:37.067+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:37.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:37.287+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:38.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:38.035+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:38.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:38.243+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:39.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:39.007+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:39.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:39.231+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:40.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:40.051+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:40.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:40.251+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:41.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:41.071+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:41.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:41.207+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:42.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:42.063+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:42.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:42.215+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:43.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:43.095+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:43.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:43.195+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:44.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:44.127+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:44.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:44.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:45.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:45.107+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:45.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:45.255+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:46.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:46.115+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:46.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:46.283+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:47.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:47.122+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:47.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:47.310+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:48.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:48.094+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:48.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:48.354+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:49.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:49.110+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:49.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:49.314+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:50.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:50.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:50.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:50.334+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:51.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:51.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:51.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:51.314+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:52.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:52.030+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:52.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:52.310+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:53.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:53.030+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:53.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:53.282+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:54.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:54.046+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:54.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:54.278+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:55.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:55.050+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:55.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:55.282+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:56.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:56.030+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:56.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:56.246+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:57.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:57.070+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:57.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:57.226+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:58.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:58.110+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:58.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:58.210+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:59.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:19:59.130+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:19:59.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:19:59.258+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:00.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:00.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:00.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:00.218+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:01.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:01.118+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:01.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:01.206+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:02.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:02.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:02.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:02.162+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:03.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:03.090+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:03.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:03.206+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:04.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:04.082+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:04.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:04.202+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:05.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:05.034+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:05.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:05.234+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:06.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:06.054+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:06.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:06.274+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:07.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:07.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:07.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:07.266+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:08.070 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:08.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:08.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:08.250+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:09.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:09.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:09.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:09.226+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:10.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:10.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:10.266 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:10.262+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:11.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:11.058+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:11.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:11.230+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:12.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:12.046+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:12.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:12.274+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:13.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:13.014+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:13.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:13.242+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:14.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:14.050+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:14.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:14.218+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:15.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:15.078+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:15.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:15.222+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:16.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:16.094+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:16.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:16.226+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:17.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:17.102+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:17.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:17.198+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:18.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:18.098+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:18.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:18.238+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:19.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:19.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:19.267 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:19.262+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:20.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:20.034+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:20.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:20.290+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:21.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:21.010+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:21.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:21.314+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:22.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:22.018+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:22.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:22.358+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:23.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:23.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:23.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:23.390+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:24.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:24.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:24.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:24.390+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:25.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:25.114+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:25.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:25.394+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:26.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:26.146+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:26.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:26.358+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:27.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:27.166+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:27.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:27.342+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:28.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:28.146+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:28.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:28.310+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:29.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:29.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:29.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:29.338+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:30.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:30.122+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:30.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:30.386+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:31.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:31.166+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:31.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:31.402+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:32.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:32.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:32.377 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:32.374+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:33.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:33.190+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:33.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:33.394+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:34.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:34.146+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:34.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:34.406+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:35.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:35.178+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:35.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:35.398+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:36.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:36.210+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:36.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:36.362+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:37.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:37.190+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:37.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:37.322+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:38.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:38.157+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:38.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:38.281+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:39.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:39.205+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:39.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:39.317+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:40.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:40.189+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:40.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:40.325+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.10580.0:2723 3.1 3:b56b769d:test1::rbd_data.2951ff622ce2.0000000000000020:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:41.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:41.173+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:41.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:41.313+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:42.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:42.149+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:42.335 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:42.329+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:43.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:43.121+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:43.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:43.361+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:44.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:44.169+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:44.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:44.385+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:45.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:45.193+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:45.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:45.425+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:46.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:46.185+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:46.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:46.405+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:47.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:47.157+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:47.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:47.457+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:48.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:48.133+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:48.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:48.477+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:49.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:49.149+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:49.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:49.445+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:50.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:50.181+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:50.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:50.457+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:51.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:51.185+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:51.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:51.429+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:52.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:52.149+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:52.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:52.461+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:53.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:53.185+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:53.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:53.425+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:54.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:54.157+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:54.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:54.457+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:55.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:55.145+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:55.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:55.449+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:56.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:56.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:56.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:56.457+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:57.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:57.205+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:57.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:57.429+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:58.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:58.229+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:58.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:58.413+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:59.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:20:59.229+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:20:59.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:20:59.425+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:00.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:00.221+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:00.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:00.457+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:01.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:01.241+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:01.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:01.493+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:02.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:02.229+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:02.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:02.529+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:03.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:03.281+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:03.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:03.573+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:04.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:04.317+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:04.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:04.569+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:05.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:05.305+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:05.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:05.613+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:06.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:06.345+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:06.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:06.617+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:07.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:07.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:07.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:07.649+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:08.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:08.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:08.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:08.633+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:09.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:09.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:09.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:09.673+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:10.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:10.385+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:10.683 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:10.677+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:11.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:11.377+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:11.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:11.693+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:12.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:12.353+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:12.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:12.697+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:13.358 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:13.353+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:13.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:13.673+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:14.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:14.397+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:14.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:14.717+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:15.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:15.397+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:15.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:15.693+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:16.366 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:16.361+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:16.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:16.737+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:17.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:17.409+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:17.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:17.777+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:18.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:18.437+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:18.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:18.825+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:19.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:19.401+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:19.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:19.777+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:20.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:20.405+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:20.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:20.781+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:21.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:21.409+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:21.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:21.793+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:22.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:22.389+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:22.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:22.761+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:23.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:23.373+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:23.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:23.753+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:24.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:24.349+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:24.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:24.721+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:25.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:25.345+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:25.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:25.741+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:26.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:26.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:26.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:26.765+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:27.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:27.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:27.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:27.769+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:28.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:28.325+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:28.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:28.820+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:29.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:29.296+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:29.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:29.856+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:30.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:30.336+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:30.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:30.872+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:31.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:31.316+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:31.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:31.844+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:32.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:32.300+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:32.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:32.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:33.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:33.352+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:33.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:33.808+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:34.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:34.388+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:34.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:34.844+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:35.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:35.416+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:35.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:35.856+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:36.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:36.388+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:36.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:36.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:37.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:37.424+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:37.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:37.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:38.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:38.468+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:38.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:38.800+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:39.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:39.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:39.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:39.852+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:40.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:40.444+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:40.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:40.884+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:41.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:41.460+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:41.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:41.844+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:42.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:42.476+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:42.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:42.888+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:43.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:43.524+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:43.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:43.860+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:44.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:44.532+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:44.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:44.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:45.538 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:45.532+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:45.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:45.896+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:46.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:46.560+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:46.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:46.948+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:47.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:47.608+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:47.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:47.968+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:48.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:48.652+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:48.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:48.992+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:49.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:49.648+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:49.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:49.964+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:50.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:50.616+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:50.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:50.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:51.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:51.624+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:51.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:51.980+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:52.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:52.664+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:52.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:52.948+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:53.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:53.664+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:53.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:53.912+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:54.667 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:54.660+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:54.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:54.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:55.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:55.704+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:55.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:55.884+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:56.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:56.752+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:56.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:56.920+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:57.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:57.736+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:57.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:57.888+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:58.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:58.704+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:58.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:58.868+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:59.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:21:59.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:21:59.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:21:59.852+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:00.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:00.704+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:00.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:00.856+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:01.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:01.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:01.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:01.832+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:02.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:02.724+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:02.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:02.880+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:03.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:03.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:03.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:03.840+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:04.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:04.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:04.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:04.800+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:05.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:05.768+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:05.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:05.776+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:06.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:06.732+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:06.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:06.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:07.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:07.696+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:07.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:07.752+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:08.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:08.748+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:08.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:08.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:09.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:09.712+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:09.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:09.768+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:10.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:10.676+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:10.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:10.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:11.691 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:11.688+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:11.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:11.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:12.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:12.700+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:12.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:12.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:13.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:13.720+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:13.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:13.784+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:14.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:14.700+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:14.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:14.816+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:15.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:15.688+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:15.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:15.852+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:16.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:16.652+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:16.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:16.864+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:17.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:17.636+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:17.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:17.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:18.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:18.668+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:18.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:18.836+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:19.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:19.623+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:19.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:19.851+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:20.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:20.607+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:20.907 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:20.903+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:21.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:21.607+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:21.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:21.875+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:22.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:22.583+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:22.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:22.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:23.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:23.555+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:23.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:23.943+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:24.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:24.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:24.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:24.919+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:25.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:25.571+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:25.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:25.899+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:26.545 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:26.543+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:26.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:26.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:27.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:27.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:27.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:27.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:28.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:28.551+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:28.907 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:28.903+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:29.587 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:29.583+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:29.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:29.931+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:30.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:30.543+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:30.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:30.943+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:31.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:31.535+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:31.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:31.915+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:32.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:32.579+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:32.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:32.887+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:33.588 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:33.587+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:33.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:33.871+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:34.606 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:34.603+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:34.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:34.891+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:35.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:35.643+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:35.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:35.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:36.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:36.659+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:36.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:36.955+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:37.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:37.667+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:37.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:37.975+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:38.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:38.691+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:38.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:38.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:39.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:39.727+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:39.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:39.975+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:40.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:40.691+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:40.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:40.967+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:41.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:41.687+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:42.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:42.007+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:42.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:42.667+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:42.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:42.963+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:43.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:43.635+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:43.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:43.915+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:44.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:44.603+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:44.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:44.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:45.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:45.591+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:45.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:45.979+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:46.591 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:46.587+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:47.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:47.015+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:47.557 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:47.555+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:48.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:48.059+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:48.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:48.599+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:49.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:49.043+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:49.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:49.627+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:50.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:50.087+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:50.648 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:50.643+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:51.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:51.111+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:51.603 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:51.599+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:52.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:52.139+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:52.587 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:52.583+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:53.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:53.147+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:53.571 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:53.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:54.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:54.171+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:54.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:54.611+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:55.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:55.219+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:55.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:55.595+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:56.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:56.247+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:56.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:56.631+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:57.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:57.207+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:57.623 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:57.619+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:58.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:58.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:58.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:58.647+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:59.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:22:59.215+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:22:59.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:22:59.651+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:00.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:00.175+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:00.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:00.695+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:01.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:01.171+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:01.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:01.735+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:02.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:02.219+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:02.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:02.719+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:03.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:03.215+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:03.750 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:03.747+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:04.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:04.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:04.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:04.735+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:05.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:05.211+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:05.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:05.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:06.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:06.235+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:06.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:06.727+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:07.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:07.199+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:07.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:07.699+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:08.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:08.235+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:08.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:08.671+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:09.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:09.282+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:09.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:09.662+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:10.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:10.242+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:10.656 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:10.654+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:11.224 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:11.222+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:11.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:11.654+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:12.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:12.202+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:12.635 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:12.630+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:13.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:13.162+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:13.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:13.606+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:14.132 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:14.126+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:14.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:14.570+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:15.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:15.162+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:15.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:15.574+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:16.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:16.134+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:16.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:16.534+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:17.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:17.170+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:17.499 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:17.494+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:18.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:18.214+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:18.542 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:18.538+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:19.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:19.218+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:19.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:19.574+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:20.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:20.190+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:20.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:20.598+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:21.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:21.182+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:21.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:21.634+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:22.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:22.150+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:22.640 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:22.634+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:23.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:23.142+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:23.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:23.590+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:24.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:24.150+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:24.563 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:24.558+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:25.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:25.198+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:25.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:25.570+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:26.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:26.158+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:26.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:26.530+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:27.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:27.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:27.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:27.526+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:28.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:28.102+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:28.568 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:28.566+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:29.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:29.142+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:29.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:29.550+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:30.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:30.138+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:30.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:30.518+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:31.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:31.154+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:31.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:31.514+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:32.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:32.182+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:32.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:32.494+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:33.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:33.202+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:33.515 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:33.510+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:34.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:34.162+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:34.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:34.494+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:35.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:35.178+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:35.542 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:35.538+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:36.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:36.178+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:36.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:36.554+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:37.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:37.146+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:37.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:37.526+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:38.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:38.178+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:38.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:38.550+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:39.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:39.162+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:39.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:39.602+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:40.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:40.114+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:40.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:40.630+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:41.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:41.154+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:41.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:41.626+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:42.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:42.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:42.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:42.666+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:43.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:43.106+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:43.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:43.630+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:44.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:44.154+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:44.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:44.678+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:45.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:45.138+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:45.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:45.662+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:46.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:46.122+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:46.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:46.710+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:47.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:47.138+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:47.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:47.666+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:48.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:48.114+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:48.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:48.618+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:49.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:49.086+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:49.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:49.598+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:50.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:50.138+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:50.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:50.566+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:51.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:51.102+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:51.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:51.522+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:52.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:52.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:52.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:52.486+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:53.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:53.070+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:53.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:53.526+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:54.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:54.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:54.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:54.542+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:55.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:55.078+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:55.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:55.546+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:56.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:56.070+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:56.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:56.566+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:57.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:57.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:57.619 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:57.614+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:58.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:58.002+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:58.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:58.634+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:58.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:58.982+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:59.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:23:59.602+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:23:59.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:23:59.946+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:00.562 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:00.558+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:00.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:00.958+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:01.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:01.525+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:01.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:01.981+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:02.578 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:02.573+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:02.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:02.961+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:03.625 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:03.621+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:03.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:03.945+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:04.648 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:04.641+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:04.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:04.953+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:05.603 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:05.597+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:05.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:05.921+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:06.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:06.561+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:06.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:06.949+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:07.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:07.585+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:08.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:07.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:08.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:08.593+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:08.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:08.985+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:09.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:09.625+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:10.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:10.017+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:10.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:10.601+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:11.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:11.061+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:11.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:11.641+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:12.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:12.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:12.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:12.665+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:13.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:13.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:13.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:13.657+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:14.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:14.021+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:14.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:14.697+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:15.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:15.017+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:15.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:15.737+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:16.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:16.029+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:16.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:16.721+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:17.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:17.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:17.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:17.757+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:18.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:18.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:18.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:18.793+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:19.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:19.057+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:19.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:19.809+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:20.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:20.101+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:20.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:20.833+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:21.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:21.125+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:21.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:21.877+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:22.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:22.109+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:22.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:22.877+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:23.084 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:23.077+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:23.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:23.865+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:24.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:24.113+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:24.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:24.837+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:25.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:25.077+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:25.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:25.789+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:26.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:26.097+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:26.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:26.753+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:27.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:27.117+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:27.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:27.785+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:28.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:28.133+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:28.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:28.829+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:29.110 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:29.105+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:29.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:29.789+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:30.101 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:30.097+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:30.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:30.805+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:31.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:31.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:31.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:31.801+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:32.060 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:32.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:32.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:32.825+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:33.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:33.017+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:33.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:33.853+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:34.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:34.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:34.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:34.893+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:35.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:35.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:35.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:35.921+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:36.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:36.021+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:36.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:36.905+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:37.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:37.073+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:37.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:37.905+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:38.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:38.069+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:38.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:38.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:39.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:39.093+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:39.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:39.985+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:40.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:40.109+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:41.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:41.013+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:41.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:41.153+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:42.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:42.021+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:42.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:42.153+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:43.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:42.993+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:43.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:43.197+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:44.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:44.021+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:44.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:44.209+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:44.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:44.977+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:45.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:45.205+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:46.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:45.997+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:46.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:46.237+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:46.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:46.977+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:47.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:47.265+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:47.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:47.985+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:48.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:48.229+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:48.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:48.941+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:49.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:49.201+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:49.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:49.897+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:50.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:50.249+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:50.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:50.945+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:51.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:51.249+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:51.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:51.989+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:52.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:52.285+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:53.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:53.025+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:53.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:53.293+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:54.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:54.005+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:54.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:54.309+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:54.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:54.968+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:55.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:55.308+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:56.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:56.016+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:56.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:56.352+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:57.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:57.000+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:57.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:57.312+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:58.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:57.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:58.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:58.272+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:58.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:58.972+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:59.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:24:59.268+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:24:59.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:24:59.964+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:00.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:00.268+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:00.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:00.992+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:01.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:01.240+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:01.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:01.992+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:02.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:02.264+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:03.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:02.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:03.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:03.276+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:03.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:03.984+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:04.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:04.236+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:04.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:04.964+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:05.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:05.276+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:05.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:05.956+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:06.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:06.312+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:06.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:06.912+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:07.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:07.292+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:07.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:07.868+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:08.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:08.248+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:08.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:08.876+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:09.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:09.228+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:09.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:09.924+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:10.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:10.200+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:10.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:10.896+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:11.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:11.232+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:11.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:11.852+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:12.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:12.192+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:12.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:12.852+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:13.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:13.180+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:13.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:13.832+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:14.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:14.144+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:14.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:14.828+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:15.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:15.100+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:15.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:15.784+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:16.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:16.128+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:16.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:16.764+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:17.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:17.172+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:17.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:17.784+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:18.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:18.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:18.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:18.748+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:19.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:19.240+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:19.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:19.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:20.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:20.252+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:20.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:20.708+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:21.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:21.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:21.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:21.700+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:22.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:22.260+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:22.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:22.688+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:23.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:23.304+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:23.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:23.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:24.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:24.280+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:24.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:24.644+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:25.267 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:25.260+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:25.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:25.648+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:26.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:26.216+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:26.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:26.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:27.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:27.240+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:27.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:27.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:28.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:28.212+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:28.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:28.724+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:29.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:29.168+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:29.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:29.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:30.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:30.188+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:30.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:30.684+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:31.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:31.160+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:31.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:31.700+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:32.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:32.200+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:32.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:32.716+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:33.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:33.216+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:33.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:33.728+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:34.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:34.188+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:34.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:34.768+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:35.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:35.152+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:35.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:35.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:36.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:36.196+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:36.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:36.828+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:37.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:37.192+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:37.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:37.816+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:38.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:38.156+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:38.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:38.844+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:39.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:39.180+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:39.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:39.828+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:40.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:40.140+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:40.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:40.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:41.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:41.144+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:41.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:41.788+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:42.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:42.196+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:42.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:42.744+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:43.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:43.188+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:43.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:43.736+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:44.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:44.164+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:44.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:44.776+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:45.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:45.196+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:45.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:45.752+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:46.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:46.248+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:46.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:46.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:47.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:47.296+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:47.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:47.756+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:48.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:48.283+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:48.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:48.751+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:49.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:49.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:49.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:49.783+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:50.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:50.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:50.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:50.803+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:51.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:51.295+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:51.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:51.763+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:52.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:52.283+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:52.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:52.731+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:53.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:53.303+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:53.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:53.771+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:54.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:54.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:54.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:54.791+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:55.321 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:55.319+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:55.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:55.783+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:56.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:56.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:56.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:56.791+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:57.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:57.315+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:57.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:57.807+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:58.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:58.291+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:58.808 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:58.803+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:59.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:25:59.303+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:25:59.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:25:59.851+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:00.335 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:00.331+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:00.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:00.807+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:01.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:01.295+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:01.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:01.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:02.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:02.299+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:02.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:02.743+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:03.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:03.283+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:03.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:03.775+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:04.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:04.331+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:04.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:04.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:05.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:05.283+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:05.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:05.695+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:06.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:06.287+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:06.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:06.723+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:07.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:07.291+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:07.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:07.727+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:08.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:08.255+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:08.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:08.731+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:09.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:09.215+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:09.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:09.759+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:10.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:10.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:10.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:10.719+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:11.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:11.307+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:11.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:11.687+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:12.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:12.311+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:12.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:12.643+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:13.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:13.299+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:13.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:13.635+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:14.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:14.267+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:14.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:14.635+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:15.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:15.319+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:15.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:15.591+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:16.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:16.343+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:16.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:16.551+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:17.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:17.355+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:17.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:17.599+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:18.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:18.391+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:18.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:18.599+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:19.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:19.387+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:19.596 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:19.595+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:20.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:20.379+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:20.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:20.555+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:21.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:21.375+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:21.540 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:21.535+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:22.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:22.379+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:22.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:22.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:23.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:23.355+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:23.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:23.555+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:24.377 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:24.375+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:24.578 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:24.575+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:25.402 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:25.399+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:25.591 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:25.587+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:26.358 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:26.355+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:26.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:26.575+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:27.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:27.383+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:27.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:27.603+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:28.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:28.359+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:28.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:28.587+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:29.337 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:29.335+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:29.588 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:29.583+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:30.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:30.307+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:30.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:30.595+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:31.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:31.299+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:31.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:31.603+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:32.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:32.251+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:32.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:32.611+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:33.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:33.227+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:33.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:33.599+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:34.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:34.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:34.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:34.631+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:35.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:35.227+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:35.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:35.651+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:36.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:36.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:36.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:36.611+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:37.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:37.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:37.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:37.659+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:38.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:38.247+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:38.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:38.623+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:39.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:39.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:39.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:39.667+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:40.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:40.230+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:40.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:40.714+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:41.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:41.258+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:41.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:41.750+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:42.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:42.310+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:42.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:42.738+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:43.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:43.342+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:43.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:43.762+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:44.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:44.326+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:44.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:44.810+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:45.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:45.350+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:45.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:45.850+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:46.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:46.306+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:46.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:46.834+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:47.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:47.346+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:47.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:47.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:48.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:48.382+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:48.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:48.798+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:49.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:49.354+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:49.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:49.838+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:50.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:50.350+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:50.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:50.874+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:51.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:51.378+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:51.845 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:51.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:52.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:52.426+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:52.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:52.862+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:53.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:53.446+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:53.845 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:53.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:54.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:54.430+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:54.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:54.830+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:55.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:55.470+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:55.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:55.822+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:56.458 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:56.454+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:56.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:56.826+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:57.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:57.486+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:57.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:57.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:58.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:58.450+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:58.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:58.830+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:59.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:26:59.450+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:26:59.874 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:26:59.870+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:00.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:00.402+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:00.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:00.834+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:01.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:01.370+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:01.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:01.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:02.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:02.366+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:02.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:02.854+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:03.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:03.362+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:03.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:03.850+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:04.377 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:04.374+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:04.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:04.838+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:05.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:05.410+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:05.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:05.846+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:06.449 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:06.446+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:06.845 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:06.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:07.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:07.490+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:07.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:07.802+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:08.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:08.450+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:08.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:08.786+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:09.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:09.458+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:09.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:09.778+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:10.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:10.486+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:10.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:10.806+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:11.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:11.526+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:11.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:11.846+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:12.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:12.538+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:12.862 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:12.858+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:13.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:13.578+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:13.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:13.838+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:14.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:14.558+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:14.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:14.882+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:15.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:15.578+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:15.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:15.850+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:16.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:16.562+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:16.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:16.842+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:17.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:17.594+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:17.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:17.866+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:18.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:18.618+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:18.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:18.890+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:19.626 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:19.622+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:19.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:19.878+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:20.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:20.654+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:20.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:20.918+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:21.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:21.614+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:21.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:21.950+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:22.605 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:22.602+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:22.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:22.954+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:23.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:23.590+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:23.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:23.938+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:24.631 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:24.626+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:24.964 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:24.958+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:25.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:25.622+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:25.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:25.922+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:26.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:26.618+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:26.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:26.922+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:27.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:27.634+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:27.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:27.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:28.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:28.606+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:28.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:28.950+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:29.571 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:29.566+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:29.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:29.934+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:30.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:30.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:30.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:30.966+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:31.563 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:31.558+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:31.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:31.918+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:32.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:32.517+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:32.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:32.953+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:33.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:33.533+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:33.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:33.973+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:34.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:34.541+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:34.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:34.929+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:35.516 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:35.509+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:35.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:35.881+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:36.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:36.513+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:36.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:36.877+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:37.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:37.501+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:37.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:37.897+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:38.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:38.521+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:38.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:38.917+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:39.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:39.565+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:39.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:39.937+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:40.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:40.609+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:40.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:40.953+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:41.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:41.645+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:41.968 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:41.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:42.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:42.641+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:42.938 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:42.933+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:43.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:43.625+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:43.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:43.977+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:44.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:44.629+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:44.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:44.969+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:45.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:45.593+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:45.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:45.981+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:46.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:46.617+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:46.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:46.973+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:47.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:47.593+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:48.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:48.021+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:48.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:48.641+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:49.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:49.053+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:49.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:49.653+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:50.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:50.021+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:50.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:50.641+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:51.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:51.001+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:51.668 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:51.661+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:51.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:51.965+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:52.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:52.689+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:52.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:52.933+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:53.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:53.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:53.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:53.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:54.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:54.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:54.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:54.977+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:55.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:55.721+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:55.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:55.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:56.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:56.701+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:56.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:56.917+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:57.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:57.729+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:57.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:57.877+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:58.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:58.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:58.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:58.881+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:59.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:27:59.789+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:27:59.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:27:59.869+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:00.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:00.793+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:00.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:00.837+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:01.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:01.801+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:01.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:01.805+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:02.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:02.757+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:02.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:02.789+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:03.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:03.745+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:03.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:03.757+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:04.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:04.745+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:04.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:04.765+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:05.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:05.725+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:05.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:05.749+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:06.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:06.685+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:06.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:06.749+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:07.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:07.653+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:07.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:07.769+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:08.618 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:08.613+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:08.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:08.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:09.649 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:09.645+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:09.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:09.765+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:10.603 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:10.597+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:10.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:10.801+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:11.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:11.641+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:11.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:11.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:12.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:12.625+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:12.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:12.865+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:13.619 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:13.613+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:13.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:13.865+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:14.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:14.637+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:14.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:14.849+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:15.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:15.625+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:15.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:15.889+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:16.678 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:16.673+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:16.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:16.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:17.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:17.681+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:17.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:17.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:18.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:18.665+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:18.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:18.857+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:19.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:19.633+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:19.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:19.837+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:20.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:20.669+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:20.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:20.845+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:21.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:21.709+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:21.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:21.861+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:22.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:22.669+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:22.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:22.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:23.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:23.708+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:23.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:23.860+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:24.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:24.744+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:24.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:24.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:25.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:25.696+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:25.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:25.856+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:26.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:26.732+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:26.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:26.884+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:27.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:27.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:27.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:27.920+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:28.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:28.780+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:28.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:28.916+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:29.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:29.740+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:29.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:29.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:30.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:30.780+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:30.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:30.936+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:31.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:31.748+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:31.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:31.916+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:32.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:32.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:32.967 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:32.960+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:33.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:33.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:33.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:33.924+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:34.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:34.828+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:34.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:34.876+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:35.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:35.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:35.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:35.884+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:36.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:36.824+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:36.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:36.868+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:37.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:37.792+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:37.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:37.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:38.792 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:38.784+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:38.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:38.864+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:39.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:39.776+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:39.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:39.872+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:40.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:40.736+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:40.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:40.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:41.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:41.772+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:41.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:41.820+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:42.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:42.772+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:42.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:42.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:43.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:43.796+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:43.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:43.892+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:44.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:44.756+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:44.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:44.876+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:45.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:45.796+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:45.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:45.924+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:46.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:46.836+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:46.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:46.936+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:47.887 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:47.880+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:47.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:47.892+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:48.892 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:48.884+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:48.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:48.896+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:49.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:49.916+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:49.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:49.924+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:50.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:50.920+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:50.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:50.952+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:51.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:51.912+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:51.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:51.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:52.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:52.924+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:52.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:52.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:53.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:53.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:53.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:53.960+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:54.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:54.916+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:54.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:54.988+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:55.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:55.956+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:55.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:55.968+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:56.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:56.948+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:56.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:56.984+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:57.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:57.960+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:57.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:57.968+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:58.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:58.916+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:59.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:59.012+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:59.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:28:59.964+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:28:59.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:28:59.980+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:00.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:00.988+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:01.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:01.016+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:02.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:01.996+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:02.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:02.028+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:02.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:02.976+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:03.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:03.036+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:03.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:03.952+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:04.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:04.044+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:04.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:04.996+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:05.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:05.000+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:06.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:06.016+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:06.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:06.020+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:07.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:07.028+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:07.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:07.036+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:07.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:07.988+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:07.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:07.992+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:08.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:08.984+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:09.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:09.036+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:10.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:10.000+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:10.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:10.084+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:10.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:10.960+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:11.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:11.056+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:11.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:11.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:12.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:12.064+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:12.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:12.948+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:13.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:13.080+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:13.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:13.984+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:14.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:14.040+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:15.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:15.007+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:15.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:15.015+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:16.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:16.011+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:16.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:16.011+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:16.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:16.987+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:17.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:17.011+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:18.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:18.011+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:18.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:18.027+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:19.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:19.023+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:19.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:19.023+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:20.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:20.055+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:20.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:20.063+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:21.023 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:21.019+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:21.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:21.087+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:22.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:22.023+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:22.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:22.079+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:23.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:23.039+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:23.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:23.075+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:24.060 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:24.055+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:24.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:24.119+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:25.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:25.059+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:25.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:25.107+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:26.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:26.087+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:26.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:26.147+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:27.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:27.115+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:27.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:27.151+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:28.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:28.103+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:28.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:28.163+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:29.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:29.123+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:29.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:29.127+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:30.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:30.099+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:30.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:30.175+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:31.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:31.119+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:31.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:31.135+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:32.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:32.139+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:32.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:32.175+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:33.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:33.111+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:33.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:33.175+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:34.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:34.083+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:34.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:34.227+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:35.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:35.119+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:35.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:35.259+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:36.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:36.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:36.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:36.303+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:37.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:37.155+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:37.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:37.275+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:38.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:38.123+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:38.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:38.275+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:39.101 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:39.099+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:39.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:39.243+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:40.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:40.119+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:40.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:40.263+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:41.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:41.151+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:41.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:41.267+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:42.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:42.139+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:42.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:42.303+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:43.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:43.171+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:43.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:43.259+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:44.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:44.147+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:44.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:44.291+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:45.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:45.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:45.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:45.255+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:46.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:46.139+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:46.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:46.215+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:47.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:47.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:47.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:47.263+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:48.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:48.115+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:48.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:48.235+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:49.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:49.139+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:49.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:49.215+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:50.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:50.151+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:50.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:50.179+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:51.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:51.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:51.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:51.167+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:52.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:52.135+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:52.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:52.155+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:53.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:53.115+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:53.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:53.171+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:54.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:54.131+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:54.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:54.183+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:55.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:55.131+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:55.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:55.191+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:56.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:56.151+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:56.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:56.175+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:57.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:57.147+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:57.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:57.187+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:58.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:58.115+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:58.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:58.199+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:59.083 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:29:59.079+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:29:59.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:29:59.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:00.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:00.051+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:00.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:00.243+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:01.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:01.043+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:01.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:01.211+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:02.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:01.999+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:02.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:02.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:03.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:03.023+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:03.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:03.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:03.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:03.975+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:04.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:04.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:04.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:04.943+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:05.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:05.258+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:05.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:05.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:06.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:06.242+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:06.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:06.938+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:07.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:07.242+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:07.900 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:07.894+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:08.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:08.282+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:08.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:08.934+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:09.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:09.258+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:09.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:09.894+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:10.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:10.290+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:10.914 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:10.910+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:11.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:11.302+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:11.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:11.914+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:12.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:12.286+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:12.900 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:12.894+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:13.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:13.314+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:13.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:13.930+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:14.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:14.338+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:14.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:14.966+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:15.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:15.346+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:15.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:15.926+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:16.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:16.342+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:16.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:16.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:17.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:17.302+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:17.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:17.974+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:18.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:18.314+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:18.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:18.942+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:19.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:19.330+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:19.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:19.966+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:20.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:20.322+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:20.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:20.938+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:21.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:21.366+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:21.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:21.914+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:22.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:22.410+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:22.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:22.966+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:23.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:23.450+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:24.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:24.002+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:24.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:24.502+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:25.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:25.034+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:25.541 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:25.538+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:26.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:26.010+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:26.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:26.522+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:26.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:26.966+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:27.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:27.490+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:27.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:27.970+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:28.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:28.506+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:28.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:28.982+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:29.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:29.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:30.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:30.006+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:30.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:30.522+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:30.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:30.982+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:31.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:31.474+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:32.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:32.026+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:32.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:32.470+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:33.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:33.078+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:33.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:33.450+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:34.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:34.094+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:34.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:34.490+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:35.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:35.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:35.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:35.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:36.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:36.146+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:36.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:36.534+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:37.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:37.146+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:37.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:37.558+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:38.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:38.162+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:38.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:38.562+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:39.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:39.166+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:39.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:39.558+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:40.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:40.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:40.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:40.566+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:41.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:41.178+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:41.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:41.590+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:42.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:42.138+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:42.631 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:42.626+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:43.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:43.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:43.626 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:43.622+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:44.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:44.122+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:44.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:44.606+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:45.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:45.106+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:45.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:45.566+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:46.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:46.146+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:46.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:46.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:47.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:47.130+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:47.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:47.558+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:48.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:48.130+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:48.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:48.582+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:49.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:49.158+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:49.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:49.582+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:50.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:50.158+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:50.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:50.574+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:51.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:51.130+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:51.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:51.622+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:52.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:52.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:52.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:52.602+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:53.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:53.150+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:53.602 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:53.598+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:54.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:54.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:54.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:54.558+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:55.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:55.098+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:55.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:55.526+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:56.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:56.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:56.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:56.534+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:57.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:57.158+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:57.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:57.574+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:58.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:58.198+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:58.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:58.586+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:59.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:30:59.246+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:30:59.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:30:59.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:00.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:00.246+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:00.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:00.534+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:01.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:01.210+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:01.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:01.542+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:02.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:02.230+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:02.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:02.570+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:03.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:03.234+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:03.578 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:03.574+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:04.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:04.218+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:04.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:04.586+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:05.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:05.206+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:05.595 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:05.590+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:06.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:06.190+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:06.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:06.617+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:07.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:07.209+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:07.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:07.601+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:08.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:08.249+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:08.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:08.649+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:09.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:09.273+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:09.672 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:09.665+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:10.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:10.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:10.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:10.685+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:11.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:11.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:11.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:11.693+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:12.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:12.281+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:12.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:12.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:13.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:13.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:13.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:13.761+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:14.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:14.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:14.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:14.809+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:15.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:15.297+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:15.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:15.849+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:16.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:16.333+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:16.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:16.837+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:17.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:17.333+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:17.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:17.797+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:18.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:18.285+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:18.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:18.841+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:19.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:19.317+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:19.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:19.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:20.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:20.317+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:20.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:20.881+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:21.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:21.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:21.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:21.869+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:22.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:22.365+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:22.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:22.889+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:23.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:23.329+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:23.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:23.933+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:24.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:24.357+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:24.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:24.901+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:25.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:25.385+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:25.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:25.925+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:26.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:26.381+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:26.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:26.881+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:27.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:27.421+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:27.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:27.849+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:28.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:28.469+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:28.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:28.893+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:29.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:29.429+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:29.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:29.941+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:30.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:30.441+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:30.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:30.897+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:31.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:31.489+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:31.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:31.865+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:32.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:32.453+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:32.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:32.853+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:33.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:33.489+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:33.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:33.869+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:34.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:34.453+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:34.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:34.861+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:35.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:35.481+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:35.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:35.901+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:36.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:36.505+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:36.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:36.893+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:37.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:37.497+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:37.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:37.905+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:38.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:38.525+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:38.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:38.929+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:39.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:39.549+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:39.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:39.893+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:40.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:40.553+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:40.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:40.849+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:41.535 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:41.529+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:41.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:41.809+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:42.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:42.501+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:42.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:42.845+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:43.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:43.481+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:43.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:43.813+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:44.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:44.457+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:44.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:44.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:45.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:45.429+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:45.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:45.785+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:46.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:46.449+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:46.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:46.797+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:47.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:47.485+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:47.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:47.789+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:48.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:48.437+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:48.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:48.825+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:49.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:49.393+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:49.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:49.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:50.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:50.353+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:50.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:50.841+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:51.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:51.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:51.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:51.845+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:52.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:52.337+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:52.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:52.857+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:53.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:53.353+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:53.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:53.821+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:54.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:54.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:54.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:54.805+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:55.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:55.369+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:55.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:55.769+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:56.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:56.357+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:56.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:56.753+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:57.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:57.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:57.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:57.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:58.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:58.317+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:58.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:58.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:59.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:31:59.293+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:31:59.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:31:59.861+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:00.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:00.257+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:00.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:00.881+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:01.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:01.277+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:01.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:01.853+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:02.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:02.297+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:02.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:02.821+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:03.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:03.261+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:03.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:03.777+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:04.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:04.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:04.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:04.761+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:05.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:05.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:05.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:05.768+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:06.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:06.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:06.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:06.772+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:07.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:07.240+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:07.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:07.744+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:08.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:08.288+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:08.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:08.748+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:09.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:09.272+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:09.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:09.736+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:10.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:10.260+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:10.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:10.704+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:11.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:11.292+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:11.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:11.732+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:12.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:12.344+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:12.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:12.728+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:13.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:13.328+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:13.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:13.720+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:14.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:14.308+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:14.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:14.688+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:15.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:15.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:15.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:15.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:16.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:16.276+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:16.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:16.768+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:17.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:17.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:17.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:17.816+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:18.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:18.320+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:18.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:18.868+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:19.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:19.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:19.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:19.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:20.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:20.244+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:20.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:20.872+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:21.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:21.240+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:21.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:21.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:22.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:22.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:22.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:22.876+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:23.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:23.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:23.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:23.880+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:24.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:24.272+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:24.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:24.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:25.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:25.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:25.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:25.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:26.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:26.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:26.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:26.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:27.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:27.256+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:27.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:27.788+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:28.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:28.228+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:28.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:28.756+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:29.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:29.264+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:29.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:29.776+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:30.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:30.252+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:30.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:30.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:31.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:31.220+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:31.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:31.764+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:32.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:32.176+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:32.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:32.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:33.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:33.164+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:33.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:33.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:34.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:34.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:34.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:34.844+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:35.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:35.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:35.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:35.896+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:36.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:36.192+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:36.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:36.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:37.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:37.152+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:37.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:37.876+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:38.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:38.184+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:38.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:38.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:39.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:39.176+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:39.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:39.928+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:40.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:40.136+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:40.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:40.928+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:41.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:41.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:41.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:41.944+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:42.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:42.120+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:42.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:42.940+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:43.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:43.140+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:43.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:43.912+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:44.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:44.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:44.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:44.960+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:45.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:45.176+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:45.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:45.992+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:46.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:46.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:46.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:46.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:47.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:47.092+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:47.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:47.952+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:48.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:48.052+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:48.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:48.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:49.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:49.084+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:49.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:49.888+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:50.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:50.128+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:50.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:50.900+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:51.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:51.164+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:51.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:51.916+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:52.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:52.156+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:52.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:52.948+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:53.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:53.164+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:53.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:53.996+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:54.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:54.124+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:54.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:54.984+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:55.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:55.160+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:56.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:56.004+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:56.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:56.176+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:57.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:57.024+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:57.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:57.152+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:58.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:58.044+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:58.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:58.112+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:59.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:32:59.080+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:32:59.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:32:59.156+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:00.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:00.068+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:00.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:00.132+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:01.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:01.096+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:01.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:01.172+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:02.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:02.052+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:02.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:02.192+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:03.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:03.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:03.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:03.171+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:04.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:04.067+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:04.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:04.155+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:05.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:05.047+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:05.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:05.159+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:06.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:06.087+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:06.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:06.203+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:07.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:07.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:07.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:07.223+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:08.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:08.039+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:08.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:08.271+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:09.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:09.079+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:09.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:09.323+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:10.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:10.099+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:10.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:10.347+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:11.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:11.111+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:11.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:11.303+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:12.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:12.095+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:12.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:12.315+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:13.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:13.119+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:13.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:13.295+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:14.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:14.107+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:14.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:14.287+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:15.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:15.063+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:15.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:15.291+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:16.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:16.063+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:16.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:16.319+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:17.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:17.043+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:17.373 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:17.371+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:18.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:17.999+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:18.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:18.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:19.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:19.027+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:19.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:19.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:20.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:20.035+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:20.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:20.383+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:21.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:21.075+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:21.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:21.359+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:22.053 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:22.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:22.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:22.335+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:23.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:23.019+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:23.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:23.343+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:23.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:23.995+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:24.373 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:24.371+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:25.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:25.027+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:25.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:25.419+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:26.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:26.047+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:26.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:26.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:27.071 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:27.067+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:27.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:27.419+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:28.101 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:28.099+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:28.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:28.439+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:29.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:29.079+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:29.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:29.491+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:30.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:30.087+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:30.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:30.503+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:31.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:31.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:31.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:31.511+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:32.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:32.035+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:32.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:32.487+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:33.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:33.007+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:33.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:33.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:34.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:34.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:34.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:34.511+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:35.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:35.007+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:35.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:35.515+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:35.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:35.971+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:36.506 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:36.503+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:36.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:36.947+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:37.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:37.551+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:37.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:37.919+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:38.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:38.523+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:38.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:38.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:39.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:39.475+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:39.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:39.895+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:40.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:40.499+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:40.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:40.915+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:41.542 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:41.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:41.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:41.867+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:42.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:42.523+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:42.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:42.859+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:43.545 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:43.543+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:43.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:43.879+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:44.524 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:44.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:44.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:44.871+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:45.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:45.499+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:45.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:45.831+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:46.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:46.491+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:46.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:46.799+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:47.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:47.463+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:47.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:47.831+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:48.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:48.435+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:48.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:48.799+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:49.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:49.431+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:49.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:49.771+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:50.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:50.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:50.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:50.779+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:51.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:51.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:51.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:51.819+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:52.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:52.435+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:52.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:52.823+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:53.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:53.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:53.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:53.779+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:54.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:54.387+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:54.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:54.787+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:55.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:55.435+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:55.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:55.815+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:56.429 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:56.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:56.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:56.823+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:57.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:57.447+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:57.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:57.791+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:58.423 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:58.419+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:58.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:58.791+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:59.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:33:59.438+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:33:59.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:33:59.806+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:00.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:00.426+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:00.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:00.826+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:01.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:01.426+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:01.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:01.850+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:02.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:02.434+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:02.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:02.886+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:03.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:03.454+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:03.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:03.874+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:04.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:04.486+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:04.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:04.854+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:05.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:05.494+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:05.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:05.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:06.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:06.526+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:06.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:06.846+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:07.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:07.570+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:07.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:07.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:08.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:08.542+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:08.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:08.862+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:09.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:09.542+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:09.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:09.846+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:10.521 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:10.518+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:10.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:10.882+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:11.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:11.546+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:11.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:11.854+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:12.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:12.594+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:12.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:12.882+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:13.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:13.590+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:13.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:13.906+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:14.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:14.638+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:14.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:14.878+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:15.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:15.690+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:15.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:15.906+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:16.679 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:16.674+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:16.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:16.954+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:17.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:17.690+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:17.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:17.962+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:18.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:18.694+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:18.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:18.922+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:19.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:19.654+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:19.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:19.938+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:20.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:20.610+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:20.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:20.954+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:21.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:21.630+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:21.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:21.938+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:22.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:22.586+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:22.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:22.934+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:23.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:23.586+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:23.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:23.894+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:24.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:24.562+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:24.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:24.942+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:25.591 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:25.586+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:25.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:25.966+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:26.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:26.574+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:26.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:26.950+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:27.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:27.586+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:27.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:27.930+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:28.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:28.542+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:28.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:28.962+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:29.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:29.546+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:29.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:29.946+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:30.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:30.526+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:30.997 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:30.994+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:31.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:31.526+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:31.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:31.986+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:32.512 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:32.510+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:32.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:32.958+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:33.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:33.514+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:33.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:33.986+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:34.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:34.502+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:35.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:35.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:35.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:35.458+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:36.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:36.014+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:36.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:36.478+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:37.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:37.002+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:37.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:37.474+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:37.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:37.970+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:38.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:38.494+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:38.982 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:38.978+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:39.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:39.458+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:39.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:39.966+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:40.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:40.486+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:41.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:41.018+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:41.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:41.518+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:42.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:41.998+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:42.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:42.474+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:42.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:42.970+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:43.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:43.478+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:43.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:43.974+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:44.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:44.522+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:44.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:44.982+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:45.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:45.526+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:46.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:46.034+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:46.545 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:46.542+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:47.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:47.082+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:47.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:47.522+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:48.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:48.094+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:48.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:48.486+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:49.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:49.134+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:49.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:49.530+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:50.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:50.138+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:50.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:50.514+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:51.176 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:51.174+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:51.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:51.522+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:52.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:52.134+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:52.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:52.550+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:53.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:53.174+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:53.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:53.542+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:54.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:54.157+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:54.561 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:54.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:55.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:55.125+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:55.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:55.569+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:56.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:56.105+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:56.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:56.617+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:57.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:57.117+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:57.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:57.653+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:58.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:58.077+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:58.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:58.685+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:59.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:34:59.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:34:59.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:34:59.669+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:00.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:00.001+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:00.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:00.621+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:00.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:00.989+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:01.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:01.609+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:02.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:02.029+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:02.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:02.561+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:03.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:03.073+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:03.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:03.513+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:04.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:04.113+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:04.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:04.485+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:05.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:05.117+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:05.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:05.481+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:06.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:06.133+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:06.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:06.465+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:07.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:07.137+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:07.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:07.453+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:08.101 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:08.097+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:08.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:08.465+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:09.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:09.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:09.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:09.445+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:10.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:10.009+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:10.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:10.413+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:11.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:11.005+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:11.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:11.381+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:12.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:12.021+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:12.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:12.429+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:12.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:12.977+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:13.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:13.445+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:13.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:13.941+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:14.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:14.457+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:14.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:14.893+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:15.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:15.489+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:15.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:15.905+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:16.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:16.473+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:16.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:16.881+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:17.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:17.489+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:17.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:17.897+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:18.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:18.525+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:18.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:18.909+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:19.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:19.553+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:19.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:19.945+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:20.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:20.593+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:20.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:20.973+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:21.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:21.597+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:22.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:22.005+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:22.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:22.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:22.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:22.965+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:23.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:23.597+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:23.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:23.937+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:24.561 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:24.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:24.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:24.969+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:25.521 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:25.517+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:25.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:25.957+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:26.563 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:26.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:27.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:26.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:27.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:27.573+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:28.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:28.017+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:28.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:28.589+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:29.017 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:29.013+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:29.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:29.593+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:29.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:29.977+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:30.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:30.565+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:30.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:30.981+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:31.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:31.533+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:31.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:31.957+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:32.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:32.497+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:32.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:32.917+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:33.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:33.485+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:33.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:33.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:34.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:34.469+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:34.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:34.873+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:35.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:35.485+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:35.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:35.893+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:36.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:36.477+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:36.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:36.881+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:37.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:37.485+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:37.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:37.853+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:38.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:38.465+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:38.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:38.861+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:39.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:39.513+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:39.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:39.845+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:40.521 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:40.517+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:40.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:40.889+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:41.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:41.473+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:41.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:41.933+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:42.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:42.453+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:42.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:42.929+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:43.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:43.497+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:43.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:43.921+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:44.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:44.473+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:44.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:44.913+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:45.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:45.437+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:45.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:45.909+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:46.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:46.457+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:46.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:46.917+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:47.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:47.485+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:47.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:47.876+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:48.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:48.444+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:48.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:48.888+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:49.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:49.476+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:49.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:49.928+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:50.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:50.456+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:50.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:50.912+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:51.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:51.508+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:51.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:51.948+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:52.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:52.500+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:52.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:52.956+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:53.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:53.512+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:53.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:53.920+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:54.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:54.528+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:54.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:54.904+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:55.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:55.496+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:55.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:55.940+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:56.538 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:56.532+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:56.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:56.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:57.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:57.492+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:57.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:57.944+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:58.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:58.468+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:58.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:58.916+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:59.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:35:59.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:35:59.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:35:59.968+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:00.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:00.504+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:00.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:00.924+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:01.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:01.464+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:01.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:01.876+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:02.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:02.472+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:02.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:02.872+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:03.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:03.432+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:03.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:03.860+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:04.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:04.428+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:04.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:04.872+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:05.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:05.416+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:05.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:05.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:06.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:06.444+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:06.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:06.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:07.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:07.464+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:07.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:07.816+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:08.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:08.484+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:08.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:08.772+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:09.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:09.508+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:09.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:09.748+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:10.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:10.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:10.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:10.736+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:11.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:11.440+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:11.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:11.724+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:12.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:12.456+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:12.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:12.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:13.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:13.500+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:13.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:13.696+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:14.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:14.524+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:14.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:14.704+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:15.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:15.528+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:15.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:15.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:16.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:16.576+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:16.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:16.704+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:17.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:17.604+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:17.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:17.728+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:18.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:18.588+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:18.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:18.692+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:19.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:19.588+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:19.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:19.660+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:20.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:20.584+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:20.683 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:20.676+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:21.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:21.592+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:21.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:21.652+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:22.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:22.580+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:22.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:22.632+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:23.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:23.616+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:23.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:23.668+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:24.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:24.620+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:24.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:24.716+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:25.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:25.648+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:25.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:25.704+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:26.623 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:26.620+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:26.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:26.672+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:27.606 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:27.604+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:27.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:27.716+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:28.565 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:28.564+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:28.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:28.720+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:29.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:29.604+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:29.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:29.748+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:30.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:30.580+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:30.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:30.716+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:31.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:31.556+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:31.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:31.728+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:32.593 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:32.592+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:32.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:32.696+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:33.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:33.592+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:33.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:33.700+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:34.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:34.556+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:34.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:34.736+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:35.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:35.548+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:35.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:35.780+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:36.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:36.504+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:36.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:36.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:37.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:37.548+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:37.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:37.868+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:38.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:38.544+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:38.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:38.900+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:39.516 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:39.516+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:39.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:39.936+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:40.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:40.484+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:40.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:40.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:41.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:41.475+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:41.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:41.939+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:42.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:42.447+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:42.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:42.935+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:43.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:43.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:43.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:43.947+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:44.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:44.379+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:44.905 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:44.903+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:45.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:45.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:45.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:45.903+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:46.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:46.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:46.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:46.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:47.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:47.371+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:47.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:47.939+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:48.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:48.323+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:48.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:48.967+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:49.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:49.279+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:49.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:49.927+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:50.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:50.243+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:50.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:50.943+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:51.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:51.219+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:51.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:51.951+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:52.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:52.251+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:52.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:52.983+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:53.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:53.219+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:53.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:53.975+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:54.196 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:54.195+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:54.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:54.939+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:55.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:55.239+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:55.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:55.911+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:56.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:56.239+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:56.943 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:56.939+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:57.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:57.247+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:57.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:57.931+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:58.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:58.251+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:58.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:58.907+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:59.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:36:59.279+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:36:59.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:36:59.891+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:00.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:00.295+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:00.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:00.931+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:01.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:01.299+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:01.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:01.943+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:02.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:02.319+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:02.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:02.991+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:03.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:03.331+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:04.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:04.023+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:04.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:04.295+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:05.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:05.047+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:05.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:05.275+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:06.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:06.075+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:06.326 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:06.323+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:07.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:07.055+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:07.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:07.315+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:08.099 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:08.095+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:08.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:08.279+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:09.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:09.099+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:09.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:09.279+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:10.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:10.067+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:10.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:10.251+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:11.084 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:11.083+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:11.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:11.215+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:12.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:12.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:12.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:12.191+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:13.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:13.063+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:13.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:13.227+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:14.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:14.075+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:14.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:14.243+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:15.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:15.067+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:15.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:15.251+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:16.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:16.063+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:16.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:16.227+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:17.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:17.039+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:17.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:17.279+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:18.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:18.079+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:18.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:18.255+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:19.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:19.063+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:19.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:19.235+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:20.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:20.027+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:20.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:20.267+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:21.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:21.059+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:21.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:21.315+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:22.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:22.067+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:22.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:22.331+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:23.053 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:23.051+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:23.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:23.287+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:24.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:24.063+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:24.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:24.243+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:25.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:25.115+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:25.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:25.219+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:26.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:26.143+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:26.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:26.247+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:27.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:27.183+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:27.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:27.259+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:28.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:28.147+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:28.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:28.243+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:29.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:29.175+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:29.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:29.239+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:30.133 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:30.131+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:30.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:30.231+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:31.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:31.095+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:31.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:31.203+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:32.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:32.087+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:32.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:32.175+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:33.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:33.079+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:33.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:33.127+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:34.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:34.090+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:34.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:34.106+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:35.071 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:35.066+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:35.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:35.082+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:36.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:36.046+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:36.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:36.074+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:37.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:37.030+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:37.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:37.122+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:38.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:38.046+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:38.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:38.122+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:39.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:39.022+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:39.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:39.162+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:40.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:40.010+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:40.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:40.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:41.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:41.046+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:41.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:41.158+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:42.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:42.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:42.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:42.206+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:43.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:43.066+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:43.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:43.162+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:44.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:44.106+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:44.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:44.150+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:45.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:45.146+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:45.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:45.150+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:46.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:46.130+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:46.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:46.142+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:47.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:47.142+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:47.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:47.154+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:48.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:48.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:48.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:48.182+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:49.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:49.070+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:49.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:49.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:50.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:50.062+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:50.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:50.174+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:51.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:51.082+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:51.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:51.174+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:52.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:52.042+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:52.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:52.142+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:53.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:53.002+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:53.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:53.118+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:54.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:54.002+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:54.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:54.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:55.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:55.034+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:55.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:55.150+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:56.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:56.034+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:56.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:56.102+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:57.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:57.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:57.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:57.082+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:58.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:58.022+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:58.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:58.082+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:59.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:37:59.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:37:59.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:37:59.102+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:00.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:00.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:00.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:00.062+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:01.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:01.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:01.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:01.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:02.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:01.998+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:02.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:02.050+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:02.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:02.986+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:03.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:03.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:03.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:03.946+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:04.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:04.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:04.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:04.926+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:05.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:05.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:05.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:05.894+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:06.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:06.038+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:06.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:06.850+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:07.070 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:07.066+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:07.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:07.870+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:08.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:08.062+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:08.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:08.862+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:09.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:09.086+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:09.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:09.874+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:10.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:10.054+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:10.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:10.826+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:11.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:11.042+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:11.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:11.846+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:12.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:12.030+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:12.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:12.858+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:13.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:13.018+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:13.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:13.850+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:14.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:14.054+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:14.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:14.878+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:15.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:15.046+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:15.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:15.878+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:16.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:16.098+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:16.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:16.886+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:17.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:17.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:17.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:17.918+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:18.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:18.146+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:18.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:18.962+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:19.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:19.118+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:19.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:19.938+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:20.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:20.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:20.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:20.982+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:21.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:21.130+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:22.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:22.022+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:22.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:22.158+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:23.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:23.010+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:23.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:23.134+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:23.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:23.986+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:24.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:24.162+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:25.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:25.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:25.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:25.126+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:26.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:26.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:26.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:26.109+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:27.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:27.001+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:27.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:27.097+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:28.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:28.045+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:28.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:28.073+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:29.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:29.057+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:29.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:29.069+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:30.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:30.049+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:30.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:30.069+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:31.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:31.009+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:31.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:31.037+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:32.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:32.013+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:32.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:32.061+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:32.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:32.989+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:33.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:33.069+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:33.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:33.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:34.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:34.049+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:34.926 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:34.921+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:35.041 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:35.037+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:35.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:35.933+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:36.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:36.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:36.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:36.965+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:36.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:36.989+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:37.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:37.937+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:38.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:38.029+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:38.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:38.897+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:39.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:39.033+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:39.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:39.857+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:40.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:40.049+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:40.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:40.813+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:41.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:41.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:41.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:41.849+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:42.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:42.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:42.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:42.813+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:43.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:43.073+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:43.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:43.857+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:44.053 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:44.049+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:44.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:44.905+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:45.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:45.061+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:45.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:45.881+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:46.053 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:46.049+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:46.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:46.889+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:47.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:47.021+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:47.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:47.937+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:48.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:48.045+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:48.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:48.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:49.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:49.001+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:49.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:49.977+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:50.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:50.049+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:50.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:50.961+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:51.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:51.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:51.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:51.929+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:51.982 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:51.977+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:52.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:52.961+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:52.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:52.973+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:53.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:53.921+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:53.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:53.981+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:54.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:54.949+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:55.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:55.001+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:56.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:55.997+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:56.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:56.033+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:56.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:56.989+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:57.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:57.001+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:57.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:57.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:57.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:57.957+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:58.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:58.953+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:59.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:38:59.005+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:38:59.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:38:59.917+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:00.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:00.009+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:00.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:00.953+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:00.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:00.993+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:01.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:01.937+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:02.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:02.005+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:02.966 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:02.961+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:02.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:02.965+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:03.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:03.929+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:04.010 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:04.005+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:04.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:04.965+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:05.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:05.001+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:05.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:05.949+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:06.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:06.029+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:06.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:06.909+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:07.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:07.061+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:07.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:07.901+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:08.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:08.045+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:08.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:08.905+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:09.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:09.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:09.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:09.893+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:10.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:10.041+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:10.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:10.853+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:11.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:11.025+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:11.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:11.821+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:12.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:12.053+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:12.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:12.801+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:13.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:13.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:13.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:13.849+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:14.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:14.065+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:14.879 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:14.873+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:15.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:15.101+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:15.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:15.873+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:16.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:16.117+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:16.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:16.837+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:17.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:17.153+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:17.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:17.848+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:18.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:18.168+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:18.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:18.864+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:19.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:19.196+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:19.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:19.856+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:20.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:20.168+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:20.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:20.852+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:21.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:21.148+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:21.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:21.884+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:22.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:22.164+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:22.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:22.852+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:23.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:23.120+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:23.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:23.844+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:24.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:24.112+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:24.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:24.808+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:25.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:25.136+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:25.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:25.844+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:26.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:26.180+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:26.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:26.852+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:27.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:27.156+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:27.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:27.852+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:28.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:28.148+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:28.810 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:28.804+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:29.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:29.100+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:29.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:29.760+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:30.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:30.052+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:30.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:30.740+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:31.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:31.040+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:31.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:31.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:32.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:32.000+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:32.660 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:32.652+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:32.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:32.968+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:33.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:33.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:34.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:34.020+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:34.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:34.668+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:35.017 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:35.012+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:35.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:35.636+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:36.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:36.020+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:36.618 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:36.612+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:37.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:37.008+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:37.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:37.628+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:37.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:37.980+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:38.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:38.624+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:38.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:38.960+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:39.596 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:39.592+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:39.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:39.964+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:40.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:40.588+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:41.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:40.996+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:41.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:41.572+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:42.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:42.016+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:42.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:42.584+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:43.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:43.032+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:43.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:43.572+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:44.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:44.072+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:44.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:44.540+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:45.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:45.092+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:45.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:45.544+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:46.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:46.072+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:46.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:46.524+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:47.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:47.120+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:47.540 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:47.536+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:48.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:48.124+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:48.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:48.564+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:49.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:49.144+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:49.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:49.572+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:50.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:50.196+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:50.610 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:50.604+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:51.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:51.224+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:51.618 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:51.612+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:52.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:52.256+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:52.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:52.592+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:53.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:53.304+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:53.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:53.564+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:54.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:54.284+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:54.619 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:54.616+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:55.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:55.264+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:55.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:55.624+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:56.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:56.256+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:56.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:56.580+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:57.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:57.236+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:57.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:57.572+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:58.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:58.228+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:58.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:58.576+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:59.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:39:59.228+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:39:59.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:39:59.592+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:00.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:00.272+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:00.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:00.548+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:01.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:01.292+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:01.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:01.540+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:02.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:02.280+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:02.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:02.516+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:03.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:03.244+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:03.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:03.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:04.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:04.212+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:04.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:04.448+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:05.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:05.220+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:05.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:05.404+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:06.266 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:06.264+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:06.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:06.392+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:07.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:07.272+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:07.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:07.392+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:08.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:08.284+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:08.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:08.392+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:09.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:09.304+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:09.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:09.396+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:10.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:10.307+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:10.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:10.391+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:11.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:11.279+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:11.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:11.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:12.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:12.251+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:12.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:12.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:13.258 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:13.255+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:13.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:13.387+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:14.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:14.247+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:14.380 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:14.379+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:15.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:15.199+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:15.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:15.367+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:16.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:16.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:16.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:16.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:17.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:17.175+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:17.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:17.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:18.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:18.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:18.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:18.375+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:19.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:19.255+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:19.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:19.403+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:20.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:20.295+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:20.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:20.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:21.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:21.251+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:21.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:21.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:22.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:22.227+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:22.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:22.343+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:23.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:23.259+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:23.330 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:23.327+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:24.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:24.243+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:24.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:24.279+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:25.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:25.207+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:25.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:25.267+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:26.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:26.243+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:26.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:26.275+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:27.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:27.199+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:27.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:27.239+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:28.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:28.179+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:28.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:28.283+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:29.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:29.163+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:29.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:29.271+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:30.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:30.211+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:30.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:30.307+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:31.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:31.263+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:31.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:31.315+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:32.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:32.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:32.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:32.271+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:33.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:33.211+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:33.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:33.247+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:34.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:34.215+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:34.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:34.263+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:35.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:35.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:35.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:35.263+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:36.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:36.231+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:36.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:36.299+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:37.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:37.187+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:37.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:37.303+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:38.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:38.159+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:38.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:38.307+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:39.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:39.127+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:39.351 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:39.347+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:40.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:40.167+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:40.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:40.395+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:41.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:41.187+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:41.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:41.387+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:42.172 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:42.167+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:42.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:42.347+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:43.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:43.187+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:43.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:43.311+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:44.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:44.171+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:44.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:44.351+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:45.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:45.175+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:45.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:45.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:46.172 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:46.171+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:46.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:46.351+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:47.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:47.203+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:47.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:47.351+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:48.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:48.227+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:48.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:48.383+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:49.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:49.195+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:49.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:49.395+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:50.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:50.187+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:50.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:50.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:51.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:51.151+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:51.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:51.415+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:52.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:52.183+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:52.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:52.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:53.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:53.131+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:53.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:53.407+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:54.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:54.179+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:54.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:54.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:55.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:55.187+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:55.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:55.395+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:56.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:56.223+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:56.380 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:56.379+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:57.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:57.243+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:57.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:57.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:58.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:58.275+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:58.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:58.399+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:59.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:40:59.239+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:40:59.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:40:59.447+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:00.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:00.271+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:00.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:00.479+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:01.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:01.230+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:01.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:01.514+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:02.266 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:02.262+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:02.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:02.498+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:03.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:03.254+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:03.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:03.470+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:04.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:04.250+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:04.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:04.458+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:05.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:05.266+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:05.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:05.498+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:06.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:06.298+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:06.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:06.498+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:07.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:07.254+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:07.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:07.534+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:08.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:08.274+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:08.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:08.498+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:09.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:09.274+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:09.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:09.490+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:10.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:10.226+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:10.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:10.482+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:11.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:11.246+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:11.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:11.438+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:12.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:12.202+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:12.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:12.426+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:13.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:13.158+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:13.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:13.382+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:14.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:14.202+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:14.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:14.406+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:15.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:15.174+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:15.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:15.382+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:16.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:16.178+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:16.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:16.338+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:17.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:17.182+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:17.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:17.310+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:18.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:18.170+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:18.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:18.278+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:19.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:19.206+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:19.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:19.310+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:20.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:20.178+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:20.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:20.298+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:21.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:21.142+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:21.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:21.278+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:22.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:22.182+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:22.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:22.238+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:23.176 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:23.174+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:23.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:23.202+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:24.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:24.154+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:24.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:24.174+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:25.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:25.154+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:25.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:25.170+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:26.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:26.130+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:26.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:26.214+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:27.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:27.150+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:27.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:27.250+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:28.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:28.190+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:28.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:28.250+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:29.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:29.194+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:29.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:29.246+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:30.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:30.150+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:30.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:30.242+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:31.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:31.110+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:31.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:31.266+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:32.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:32.098+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:32.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:32.218+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:33.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:33.102+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:33.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:33.238+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:34.093 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:34.090+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:34.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:34.246+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:35.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:35.070+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:35.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:35.278+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:36.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:36.054+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:36.330 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:36.326+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:37.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:37.086+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:37.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:37.322+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:38.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:38.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:38.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:38.354+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:39.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:39.074+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:39.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:39.390+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:40.094 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:40.090+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:40.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:40.370+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:41.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:41.142+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:41.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:41.410+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:42.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:42.126+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:42.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:42.422+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:43.134 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:43.130+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:43.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:43.434+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:44.169 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:44.166+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:44.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:44.470+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:45.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:45.162+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:45.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:45.466+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:46.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:46.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:46.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:46.434+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:47.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:47.078+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:47.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:47.434+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:48.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:48.042+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:48.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:48.430+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:49.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:49.022+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:49.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:49.430+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:50.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:50.018+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:50.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:50.402+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:51.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:51.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:51.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:51.374+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:51.987 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:51.982+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:52.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:52.333+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:52.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:52.969+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:53.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:53.377+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:53.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:53.949+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:54.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:54.429+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:54.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:54.981+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:55.413 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:55.409+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:55.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:55.977+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:56.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:56.373+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:56.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:56.949+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:57.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:57.369+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:57.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:57.929+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:58.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:58.401+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:58.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:58.893+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:59.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:41:59.413+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:41:59.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:41:59.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:00.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:00.413+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:00.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:00.861+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:01.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:01.457+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:01.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:01.885+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:02.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:02.501+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:02.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:02.861+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:03.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:03.541+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:03.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:03.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:04.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:04.569+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:04.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:04.825+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:05.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:05.597+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:05.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:05.809+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:06.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:06.565+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:06.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:06.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:07.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:07.521+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:07.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:07.841+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:08.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:08.489+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:08.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:08.813+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:09.516 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:09.509+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:09.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:09.777+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:10.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:10.517+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:10.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:10.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:11.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:11.485+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:11.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:11.797+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:12.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:12.529+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:12.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:12.833+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:13.584 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:13.577+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:13.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:13.829+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:14.591 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:14.585+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:14.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:14.873+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:15.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:15.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:15.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:15.849+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:16.580 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:16.577+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:16.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:16.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:17.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:17.533+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:17.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:17.857+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:18.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:18.505+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:18.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:18.833+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:19.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:19.465+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:19.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:19.789+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:20.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:20.505+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:20.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:20.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:21.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:21.465+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:21.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:21.729+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:22.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:22.501+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:22.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:22.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:23.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:23.521+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:23.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:23.689+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:24.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:24.545+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:24.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:24.729+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:25.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:25.537+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:25.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:25.701+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:26.564 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:26.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:26.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:26.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:27.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:27.545+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:27.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:27.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:28.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:28.529+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:28.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:28.737+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:29.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:29.497+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:29.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:29.745+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:30.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:30.513+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:30.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:30.797+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:31.543 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:31.537+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:31.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:31.817+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:32.563 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:32.557+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:32.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:32.805+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:33.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:33.585+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:33.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:33.761+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:34.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:34.561+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:34.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:34.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:35.568 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:35.561+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:35.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:35.685+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:36.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:36.553+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:36.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:36.705+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:37.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:37.505+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:37.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:37.753+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:38.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:38.497+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:38.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:38.725+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:39.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:39.521+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:39.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:39.773+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:40.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:40.549+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:40.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:40.733+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:41.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:41.581+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:41.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:41.689+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:42.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:42.609+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:42.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:42.669+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:43.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:43.632+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:43.683 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:43.676+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:44.632 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:44.628+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:44.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:44.728+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:45.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:45.600+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:45.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:45.720+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:46.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:46.576+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:46.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:46.764+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:47.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:47.552+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:47.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:47.796+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:48.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:48.544+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:48.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:48.760+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:49.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:49.532+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:49.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:49.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:50.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:50.484+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:50.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:50.700+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:51.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:51.512+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:51.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:51.748+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:52.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:52.520+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:52.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:52.780+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:53.557 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:53.552+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:53.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:53.740+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:54.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:54.508+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:54.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:54.788+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:55.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:55.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:55.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:55.816+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:56.534 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:56.528+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:56.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:56.792+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:57.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:57.484+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:57.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:57.808+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:58.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:58.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:58.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:58.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:59.534 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:42:59.528+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:42:59.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:42:59.792+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:00.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:00.508+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:00.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:00.840+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:01.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:01.472+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:01.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:01.836+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:02.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:02.504+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:02.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:02.872+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:03.468 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:03.460+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:03.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:03.908+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:04.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:04.432+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:04.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:04.916+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:05.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:05.476+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:05.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:05.912+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:06.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:06.472+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:06.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:06.892+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:07.429 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:07.424+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:07.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:07.884+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:08.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:08.468+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:08.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:08.848+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:09.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:09.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:09.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:09.868+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:10.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:10.524+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:10.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:10.828+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:11.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:11.516+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:11.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:11.860+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:12.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:12.544+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:12.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:12.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:13.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:13.588+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:13.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:13.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:14.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:14.568+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:14.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:14.828+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:15.564 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:15.560+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:15.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:15.852+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:16.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:16.520+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:16.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:16.824+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:17.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:17.476+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:17.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:17.828+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:18.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:18.512+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:18.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:18.784+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:19.524 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:19.520+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:19.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:19.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:20.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:20.524+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:20.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:20.760+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:21.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:21.500+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:21.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:21.780+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:22.534 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:22.532+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:22.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:22.760+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:23.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:23.500+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:23.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:23.752+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:24.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:24.492+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:24.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:24.756+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:25.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:25.524+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:25.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:25.712+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:26.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:26.492+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:26.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:26.748+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:27.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:27.452+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:27.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:27.736+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:28.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:28.464+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:28.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:28.788+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:29.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:29.468+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:29.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:29.816+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:30.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:30.432+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:30.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:30.860+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:31.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:31.460+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:31.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:31.844+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:32.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:32.456+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:32.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:32.804+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:33.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:33.480+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:33.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:33.816+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:34.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:34.487+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:34.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:34.787+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:35.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:35.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:35.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:35.811+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:36.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:36.451+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:36.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:36.823+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:37.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:37.495+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:37.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:37.811+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:38.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:38.471+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:38.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:38.811+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:39.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:39.483+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:39.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:39.787+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:40.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:40.479+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:40.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:40.835+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:41.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:41.443+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:41.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:41.855+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:42.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:42.459+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:42.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:42.839+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:43.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:43.427+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:43.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:43.807+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:44.468 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:44.467+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:44.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:44.759+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:45.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:45.503+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:45.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:45.747+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:46.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:46.483+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:46.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:46.739+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:47.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:47.531+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:47.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:47.775+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:48.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:48.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:48.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:48.759+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:49.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:49.579+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:49.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:49.747+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:50.587 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:50.583+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:50.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:50.751+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:51.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:51.631+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:51.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:51.731+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:52.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:52.607+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:52.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:52.703+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:53.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:53.579+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:53.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:53.675+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:54.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:54.543+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:54.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:54.651+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:55.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:55.535+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:55.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:55.615+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:56.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:56.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:56.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:56.619+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:57.568 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:57.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:57.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:57.631+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:58.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:58.567+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:58.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:58.663+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:59.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:43:59.523+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:43:59.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:43:59.711+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:00.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:00.499+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:00.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:00.683+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:01.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:01.483+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:01.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:01.647+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:02.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:02.483+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:02.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:02.679+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:03.524 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:03.523+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:03.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:03.679+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:04.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:04.487+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:04.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:04.635+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:05.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:05.511+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:05.626 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:05.623+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:06.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:06.527+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:06.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:06.579+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:07.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:07.483+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:07.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:07.559+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:08.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:08.491+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:08.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:08.591+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:09.543 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:09.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:09.586 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:09.583+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:10.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:10.587+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:10.619 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:10.615+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:11.543 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:11.539+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:11.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:11.607+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:12.521 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:12.519+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:12.615 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:12.611+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:13.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:13.479+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:13.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:13.611+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:14.456 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:14.451+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:14.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:14.659+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:15.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:15.483+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:15.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:15.635+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:16.455 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:16.451+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:16.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:16.679+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:17.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:17.431+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:17.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:17.727+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:18.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:18.395+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:18.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:18.747+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:19.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:19.383+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:19.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:19.787+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:20.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:20.363+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:20.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:20.751+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:21.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:21.367+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:21.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:21.735+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:22.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:22.367+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:22.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:22.751+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:23.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:23.331+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:23.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:23.779+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:24.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:24.379+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:24.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:24.771+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:25.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:25.351+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:25.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:25.750+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:26.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:26.310+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:26.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:26.766+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:27.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:27.350+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:27.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:27.770+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:28.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:28.394+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:28.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:28.802+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:29.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:29.426+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:29.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:29.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:30.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:30.470+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:30.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:30.802+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:31.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:31.434+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:31.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:31.834+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:32.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:32.462+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:32.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:32.794+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:33.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:33.434+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:33.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:33.790+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:34.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:34.454+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:34.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:34.806+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:35.429 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:35.426+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:35.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:35.794+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:36.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:36.378+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:36.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:36.830+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:37.380 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:37.378+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:37.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:37.850+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:38.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:38.386+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:38.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:38.822+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:39.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:39.370+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:39.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:39.862+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:40.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:40.350+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:40.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:40.822+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:41.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:41.334+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:41.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:41.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:42.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:42.298+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:42.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:42.834+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:43.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:43.294+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:43.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:43.850+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:44.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:44.250+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:44.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:44.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:45.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:45.262+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:45.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:45.818+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:46.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:46.282+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:46.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:46.866+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:47.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:47.250+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:47.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:47.838+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:48.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:48.238+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:48.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:48.890+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:49.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:49.282+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:49.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:49.930+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:50.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:50.318+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:50.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:50.958+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:51.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:51.306+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:51.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:51.966+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:52.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:52.334+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:52.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:52.946+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:53.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:53.326+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:53.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:53.958+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:54.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:54.350+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:54.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:54.942+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:55.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:55.362+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:55.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:55.914+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:56.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:56.402+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:56.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:56.958+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:57.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:57.374+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:58.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:57.994+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:58.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:58.406+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:58.966 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:44:58.962+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:44:59.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:44:59.378+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:00.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:00.014+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:00.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:00.422+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:01.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:01.026+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:01.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:01.426+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:01.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:01.982+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:02.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:02.454+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:03.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:03.018+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:03.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:03.410+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:04.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:04.058+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:04.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:04.402+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:05.093 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:05.090+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:05.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:05.382+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:06.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:06.058+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:06.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:06.378+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:07.071 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:07.066+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:07.423 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:07.418+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:08.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:08.082+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:08.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:08.382+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:09.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:09.086+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:09.373 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:09.370+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:10.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:10.046+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:10.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:10.354+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:11.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:11.090+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:11.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:11.370+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:12.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:12.134+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:12.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:12.410+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:13.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:13.118+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:13.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:13.378+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:14.128 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:14.126+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:14.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:14.370+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:15.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:15.146+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:15.335 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:15.330+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:16.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:16.182+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:16.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:16.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:17.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:17.197+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:17.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:17.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:18.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:18.185+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:18.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:18.249+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:19.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:19.141+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:19.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:19.213+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:20.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:20.193+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:20.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:20.209+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:21.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:21.209+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:21.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:21.233+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:22.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:22.237+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:22.283 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:22.277+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:23.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:23.281+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:23.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:23.285+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:24.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:24.249+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:24.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:24.305+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:25.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:25.237+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:25.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:25.265+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:26.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:26.221+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:26.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:26.293+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:27.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:27.173+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:27.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:27.269+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:28.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:28.165+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:28.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:28.281+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:29.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:29.149+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:29.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:29.289+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:30.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:30.173+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:30.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:30.305+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:31.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:31.149+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:31.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:31.309+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:32.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:32.189+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:32.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:32.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:33.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:33.237+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:33.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:33.277+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:34.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:34.261+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:34.326 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:34.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:35.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:35.297+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:35.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:35.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:36.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:36.249+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:36.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:36.325+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:37.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:37.217+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:37.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:37.325+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:38.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:38.201+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:38.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:38.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:39.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:39.209+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:39.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:39.393+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:40.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:40.233+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:40.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:40.357+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:41.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:41.257+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:41.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:41.365+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:42.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:42.265+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:42.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:42.373+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:43.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:43.241+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:43.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:43.329+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:44.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:44.229+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:44.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:44.365+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:45.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:45.193+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:45.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:45.321+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:46.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:46.189+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:46.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:46.301+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:47.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:47.205+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:47.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:47.309+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:48.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:48.173+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:48.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:48.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:49.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:49.185+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:49.335 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:49.329+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:50.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:50.225+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:50.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:50.353+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:51.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:51.233+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:51.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:51.317+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:52.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:52.261+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:52.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:52.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:53.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:53.269+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:53.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:53.329+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:54.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:54.309+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:54.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:54.365+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:55.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:55.321+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:55.337 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:55.333+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:56.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:56.341+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:56.373 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:56.369+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:57.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:57.313+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:57.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:57.401+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:58.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:58.281+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:58.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:58.449+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:59.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:45:59.297+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:45:59.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:45:59.489+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:00.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:00.253+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:00.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:00.505+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:01.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:01.205+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:01.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:01.525+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:02.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:02.193+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:02.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:02.477+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:03.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:03.201+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:03.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:03.453+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:04.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:04.177+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:04.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:04.405+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:05.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:05.209+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:05.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:05.385+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:06.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:06.229+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:06.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:06.357+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:07.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:07.208+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:07.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:07.356+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:08.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:08.244+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:08.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:08.336+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:09.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:09.204+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:09.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:09.292+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:10.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:10.232+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:10.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:10.280+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:11.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:11.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:11.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:11.304+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:12.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:12.272+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:12.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:12.288+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:13.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:13.252+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:13.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:13.260+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:14.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:14.240+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:14.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:14.268+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:15.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:15.244+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:15.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:15.284+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:16.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:16.272+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:16.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:16.280+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:17.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:17.248+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:17.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:17.288+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:18.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:18.216+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:18.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:18.316+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:19.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:19.204+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:19.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:19.340+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:20.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:20.204+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:20.386 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:20.380+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:21.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:21.192+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:21.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:21.356+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:22.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:22.232+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:22.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:22.324+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:23.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:23.284+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:23.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:23.284+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:24.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:24.268+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:24.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:24.300+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:25.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:25.220+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:25.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:25.332+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:26.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:26.204+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:26.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:26.364+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:27.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:27.192+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:27.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:27.372+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:28.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:28.232+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:28.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:28.392+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:29.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:29.284+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:29.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:29.372+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:30.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:30.328+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:30.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:30.400+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:31.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:31.368+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:31.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:31.428+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:32.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:32.408+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:32.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:32.412+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:33.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:33.436+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:33.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:33.452+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:34.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:34.400+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:34.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:34.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:35.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:35.420+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:35.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:35.488+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:36.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:36.376+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:36.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:36.460+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:37.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:37.352+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:37.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:37.504+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:38.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:38.384+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:38.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:38.480+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:39.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:39.424+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:39.535 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:39.528+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:40.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:40.404+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:40.553 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:40.548+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:41.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:41.452+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:41.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:41.600+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:42.421 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:42.416+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:42.625 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:42.620+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:43.418 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:43.412+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:43.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:43.628+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:44.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:44.380+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:44.632 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:44.628+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:45.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:45.368+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:45.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:45.640+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:46.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:46.360+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:46.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:46.692+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:47.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:47.316+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:47.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:47.704+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:48.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:48.308+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:48.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:48.740+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:49.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:49.352+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:49.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:49.704+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:50.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:50.320+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:50.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:50.664+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:51.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:51.352+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:51.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:51.636+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:52.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:52.392+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:52.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:52.616+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:53.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:53.380+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:53.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:53.640+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:54.414 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:54.412+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:54.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:54.604+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:55.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:55.444+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:55.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:55.656+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:56.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:56.420+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:56.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:56.608+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:57.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:57.464+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:57.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:57.584+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:58.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:58.432+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:58.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:58.612+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:59.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:46:59.464+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:46:59.612 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:46:59.612+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:00.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:00.471+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:00.624 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:00.623+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:01.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:01.511+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:01.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:01.639+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:02.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:02.555+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:02.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:02.627+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:03.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:03.567+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:03.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:03.619+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:04.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:04.599+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:04.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:04.635+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:05.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:05.607+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:05.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:05.663+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:06.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:06.567+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:06.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:06.711+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:07.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:07.539+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:07.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:07.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:08.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:08.551+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:08.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:08.751+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:09.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:09.591+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:09.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:09.739+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:10.634 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:10.631+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:10.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:10.779+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:11.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:11.607+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:11.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:11.791+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:12.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:12.619+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:12.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:12.811+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:13.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:13.655+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:13.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:13.795+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:14.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:14.687+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:14.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:14.763+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:15.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:15.727+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:15.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:15.811+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:16.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:16.727+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:16.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:16.827+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:17.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:17.683+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:17.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:17.811+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:18.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:18.643+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:18.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:18.787+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:19.601 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:19.599+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:19.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:19.827+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:20.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:20.619+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:20.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:20.791+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:21.598 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:21.595+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:21.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:21.827+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:22.592 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:22.591+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:22.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:22.819+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:23.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:23.567+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:23.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:23.819+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:24.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:24.547+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:24.866 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:24.863+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:25.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:25.519+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:25.879 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:25.875+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:26.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:26.515+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:26.892 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:26.887+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:27.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:27.467+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:27.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:27.871+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:28.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:28.431+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:28.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:28.883+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:29.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:29.463+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:29.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:29.919+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:30.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:30.447+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:30.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:30.935+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:31.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:31.459+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:31.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:31.919+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:32.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:32.471+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:32.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:32.955+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:33.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:33.471+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:33.949 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:33.947+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:34.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:34.455+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:34.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:34.987+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:35.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:35.463+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:35.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:35.943+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:36.421 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:36.419+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:36.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:36.959+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:37.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:37.411+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:37.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:37.975+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:38.422 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:38.419+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:38.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:38.967+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:39.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:39.435+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:39.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:39.935+0000 7fdf89167700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:40.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:40.399+0000 7f6c36f0e700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:40.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:40.955+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:41.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:41.355+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:41.938 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:41.935+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:42.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:42.331+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:42.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:42.979+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:43.358 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:43.355+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:43.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:43.947+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:44.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:44.343+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:44.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:44.955+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:45.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:45.355+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:45.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:45.915+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:46.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:46.363+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:46.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:46.895+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:47.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:47.347+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:47.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:47.883+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:48.340 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:48.339+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:48.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:48.835+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:49.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:49.363+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:49.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:49.831+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:50.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:50.327+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:50.873 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:50.871+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:51.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:51.315+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:51.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:51.915+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:52.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:52.291+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:52.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:52.931+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:53.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:53.247+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:53.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:53.943+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:54.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:54.211+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:54.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:54.966+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:55.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:55.182+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:55.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:55.982+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:56.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:56.142+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:56.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:56.954+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:57.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:57.178+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:58.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:57.998+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:58.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:58.214+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:58.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:47:58.966+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:47:59.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:47:59.222+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:00.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:00.018+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:00.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:00.234+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:00.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:00.994+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:01.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:01.278+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:01.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:01.994+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:02.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:02.238+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:03.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:03.026+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:03.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:03.214+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:04.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:04.062+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:04.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:04.262+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:05.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:05.066+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:05.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:05.270+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:06.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:06.102+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:06.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:06.246+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:07.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:07.106+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:07.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:07.226+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:08.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:08.134+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:08.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:08.214+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:09.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:09.130+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:09.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:09.190+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:10.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:10.086+0000 7fdf89167700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:10.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:10.198+0000 7f6c36f0e700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:11.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:11.054+0000 7fdf89167700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:11.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:11.226+0000 7f6c36f0e700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:12.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:12.042+0000 7fdf89167700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:12.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:12.210+0000 7f6c36f0e700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:13.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:13.046+0000 7fdf89167700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:13.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:13.174+0000 7f6c36f0e700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:14.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:14.018+0000 7fdf89167700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:14.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:14.166+0000 7f6c36f0e700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:15.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:15.034+0000 7fdf89167700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:15.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:15.146+0000 7f6c36f0e700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:16.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:16.042+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:16.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:16.178+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:17.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:17.042+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:17.137 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:17.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:18.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:18.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:18.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:18.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:19.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:19.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:19.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:19.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:20.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:20.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:20.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:20.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:21.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:21.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:21.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:21.122+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:22.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:22.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:22.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:22.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:23.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:23.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:23.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:23.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:24.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:24.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:24.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:24.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:25.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:25.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:25.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:25.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:26.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:26.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:26.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:26.222+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:27.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:27.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:27.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:27.226+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:27.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:27.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:28.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:28.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:28.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:28.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:29.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:29.314+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:29.941 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:29.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:30.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:30.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:30.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:30.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:31.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:31.290+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:31.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:31.922+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:32.283 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:32.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:32.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:32.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:33.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:33.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:33.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:33.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:34.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:34.302+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:34.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:34.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:35.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:35.314+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:36.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:36.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:36.283 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:36.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:37.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:37.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:37.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:37.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:38.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:38.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:38.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:38.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:38.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:38.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:39.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:39.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:39.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:39.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:40.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:40.314+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:40.963 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:40.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:41.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:41.322+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:41.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:41.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:42.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:42.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:42.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:42.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:43.303 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:43.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:43.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:43.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:44.258 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:44.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:44.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:44.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:45.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:45.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:45.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:45.950+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:46.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:46.318+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:46.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:46.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:47.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:47.354+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:47.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:47.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:48.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:48.314+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:48.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:48.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:49.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:49.358+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:49.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:49.945+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:50.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:50.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:50.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:50.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:51.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:51.433+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:51.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:51.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:52.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:52.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:52.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:52.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:53.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:53.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:54.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:54.021+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:54.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:54.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:55.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:55.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:55.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:55.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:56.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:56.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:56.307 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:56.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:57.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:57.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:57.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:57.269+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:58.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:58.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:58.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:58.305+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:59.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:48:59.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:48:59.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:48:59.261+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:00.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:00.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:00.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:00.309+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:01.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:01.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:01.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:01.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:02.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:02.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:02.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:02.321+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:03.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:03.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:03.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:03.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:04.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:04.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:04.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:04.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:05.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:05.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:05.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:05.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:06.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:06.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:06.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:06.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:07.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:07.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:07.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:07.313+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:08.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:08.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:08.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:08.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:09.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:09.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:09.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:09.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:10.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:10.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:10.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:10.313+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:11.358 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:11.353+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:11.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:11.361+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:12.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:12.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:12.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:12.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:13.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:13.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:13.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:13.401+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:14.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:14.385+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:14.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:14.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:15.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:15.369+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:15.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:15.417+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:16.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:16.349+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:16.422 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:16.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:17.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:17.345+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:17.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:17.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:18.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:18.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:18.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:18.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:19.373 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:19.369+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:19.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:19.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:20.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:20.325+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:20.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:20.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:21.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:21.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:21.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:21.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:22.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:22.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:22.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:22.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:23.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:23.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:23.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:23.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:24.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:24.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:24.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:24.529+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:25.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:25.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:25.543 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:25.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:26.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:26.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:26.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:26.521+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:27.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:27.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:27.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:27.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:28.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:28.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:28.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:28.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:29.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:29.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:29.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:29.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:30.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:30.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:30.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:30.525+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:31.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:31.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:31.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:31.549+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:32.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:32.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:32.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:32.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:33.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:33.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:33.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:33.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:34.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:34.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:34.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:34.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:35.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:35.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:35.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:35.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:36.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:36.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:36.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:36.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:37.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:37.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:37.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:37.465+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:38.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:38.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:38.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:38.509+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:39.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:39.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:39.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:39.529+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:40.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:40.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:40.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:40.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:41.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:41.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:41.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:41.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:42.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:42.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:42.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:42.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:43.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:43.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:43.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:43.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:44.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:44.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:44.466 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:44.460+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:45.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:45.200+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:45.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:45.420+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:46.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:46.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:46.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:46.456+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:47.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:47.248+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:47.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:47.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:48.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:48.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:48.418 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:48.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:49.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:49.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:49.422 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:49.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:50.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:50.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:50.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:50.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:51.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:51.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:51.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:51.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:52.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:52.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:52.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:52.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:53.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:53.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:53.446 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:53.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:54.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:54.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:54.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:54.484+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:55.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:55.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:55.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:55.484+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:56.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:56.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:56.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:56.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:57.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:57.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:57.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:57.428+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:58.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:58.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:58.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:58.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:59.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:49:59.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:49:59.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:49:59.420+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:00.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:00.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:00.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:00.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:01.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:01.360+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:01.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:01.360+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:02.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:02.316+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:02.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:02.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:03.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:03.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:03.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:03.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:04.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:04.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:04.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:04.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:05.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:05.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:05.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:05.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:06.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:06.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:06.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:06.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:07.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:07.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:07.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:07.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:08.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:08.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:08.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:08.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:09.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:09.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:09.512 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:09.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:10.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:10.316+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:10.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:10.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:11.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:11.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:11.476 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:11.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:12.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:12.336+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:12.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:12.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:13.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:13.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:13.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:13.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:14.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:14.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:14.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:14.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:15.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:15.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:15.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:15.532+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:16.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:16.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:16.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:16.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:17.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:17.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:17.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:17.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:18.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:18.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:18.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:18.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:19.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:19.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:19.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:19.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:20.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:20.348+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:20.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:20.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:21.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:21.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:21.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:21.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:22.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:22.368+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:22.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:22.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:23.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:23.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:23.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:23.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:24.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:24.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:24.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:24.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:25.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:25.396+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:25.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:25.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:26.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:26.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:26.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:26.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:27.398 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:27.396+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:27.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:27.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:28.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:28.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:28.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:28.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:29.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:29.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:29.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:29.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:30.381 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:30.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:30.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:30.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:31.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:31.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:31.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:31.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:32.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:32.348+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:32.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:32.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:33.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:33.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:33.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:33.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:34.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:34.396+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:34.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:34.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:35.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:35.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:35.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:35.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:36.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:36.376+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:36.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:36.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:37.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:37.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:37.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:37.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:38.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:38.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:38.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:38.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:39.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:39.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:39.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:39.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:40.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:40.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:40.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:40.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:41.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:41.407+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:41.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:41.411+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:42.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:42.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:42.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:42.423+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:43.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:43.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:43.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:43.423+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:44.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:44.351+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:44.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:44.379+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:45.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:45.323+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:45.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:45.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:46.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:46.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:46.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:46.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:47.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:47.347+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:47.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:47.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:48.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:48.355+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:48.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:48.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:49.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:49.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:49.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:49.479+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:50.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:50.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:50.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:50.463+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:51.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:51.303+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:51.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:51.451+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:52.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:52.343+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:52.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:52.451+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:53.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:53.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:53.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:53.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:54.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:54.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:54.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:54.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:55.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:55.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:55.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:55.475+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:56.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:56.407+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:56.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:56.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:57.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:57.399+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:57.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:57.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:58.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:58.355+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:58.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:58.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:59.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:50:59.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:50:59.499 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:50:59.495+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:00.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:00.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:00.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:00.511+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:01.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:01.379+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:01.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:01.511+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:02.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:02.351+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:02.468 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:02.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:03.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:03.379+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:03.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:03.471+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:04.338 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:04.335+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:04.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:04.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:05.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:05.291+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:05.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:05.527+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:06.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:06.279+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:06.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:06.519+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:07.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:07.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:07.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:07.475+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:08.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:08.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:08.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:08.511+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:09.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:09.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:09.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:09.555+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:10.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:10.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:10.515 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:10.511+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:11.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:11.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:11.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:11.551+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:12.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:12.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:12.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:12.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:13.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:13.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:13.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:13.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:14.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:14.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:14.564 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:14.563+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:15.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:15.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:15.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:15.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:16.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:16.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:16.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:16.619+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:17.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:17.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:17.668 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:17.667+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:18.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:18.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:18.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:18.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:19.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:19.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:19.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:19.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:20.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:20.251+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:20.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:20.719+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:21.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:21.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:21.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:21.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:22.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:22.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:22.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:22.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:23.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:23.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:23.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:23.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:24.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:24.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:24.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:24.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:25.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:25.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:25.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:25.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:26.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:26.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:26.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:26.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:27.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:27.295+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:27.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:27.679+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:28.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:28.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:28.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:28.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:29.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:29.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:29.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:29.647+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:30.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:30.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:30.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:30.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:31.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:31.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:31.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:31.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:32.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:32.171+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:32.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:32.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:33.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:33.186+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:33.668 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:33.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:34.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:34.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:34.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:34.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:35.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:35.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:35.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:35.618+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:36.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:36.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:36.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:36.602+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:37.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:37.198+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:37.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:37.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:38.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:38.162+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:38.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:38.510+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:39.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:39.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:39.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:39.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:40.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:40.162+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:40.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:40.518+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:41.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:41.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:41.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:41.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:42.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:42.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:42.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:42.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:43.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:43.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:43.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:43.594+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:44.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:44.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:44.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:44.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:45.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:45.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:45.663 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:45.658+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:46.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:46.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:46.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:46.622+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:47.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:47.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:47.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:47.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:48.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:48.286+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:48.668 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:48.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:49.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:49.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:49.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:49.618+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:50.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:50.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:50.613 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:50.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:51.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:51.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:51.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:51.586+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:52.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:52.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:52.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:52.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:53.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:53.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:53.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:53.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:54.307 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:54.302+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:54.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:54.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:55.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:55.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:55.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:55.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:56.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:56.318+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:56.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:56.766+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:57.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:57.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:57.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:57.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:58.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:58.282+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:58.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:58.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:59.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:51:59.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:51:59.862 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:51:59.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:00.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:00.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:00.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:00.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:01.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:01.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:01.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:01.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:02.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:02.182+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:02.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:02.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:03.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:03.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:03.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:03.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:04.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:04.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:04.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:04.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:05.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:05.198+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:05.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:05.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:06.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:06.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:06.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:06.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:07.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:07.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:07.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:07.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:08.172 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:08.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:09.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:09.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:09.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:09.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:10.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:10.038+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:10.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:10.154+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:11.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:11.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:11.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:11.182+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:12.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:12.078+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:12.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:12.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:13.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:13.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:13.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:13.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:14.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:14.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:14.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:14.246+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:15.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:15.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:15.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:15.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:16.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:16.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:16.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:16.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:17.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:17.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:17.224 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:17.222+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:18.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:18.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:18.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:18.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:19.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:19.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:19.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:19.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:19.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:19.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:20.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:20.190+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:20.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:20.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:21.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:21.178+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:21.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:21.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:22.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:22.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:22.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:22.950+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:23.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:23.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:23.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:23.922+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:24.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:24.190+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:24.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:24.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:25.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:25.190+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:25.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:25.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:26.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:26.166+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:26.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:26.898+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:27.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:27.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:27.938 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:27.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:28.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:28.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:28.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:28.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:29.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:29.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:29.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:29.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:30.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:30.233+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:30.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:30.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:31.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:31.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:31.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:31.897+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:32.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:32.289+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:32.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:32.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:33.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:33.273+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:33.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:33.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:34.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:34.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:34.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:34.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:35.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:35.261+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:35.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:35.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:36.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:36.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:36.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:36.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:37.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:37.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:37.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:37.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:38.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:38.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:38.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:38.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:39.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:39.209+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:39.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:39.909+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:40.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:40.169+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:40.949 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:40.945+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:41.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:41.189+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:41.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:41.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:42.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:42.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:43.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:43.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:43.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:43.229+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:44.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:44.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:44.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:44.253+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:45.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:45.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:45.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:45.229+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:46.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:46.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:46.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:46.205+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:47.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:47.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:47.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:47.217+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:48.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:48.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:48.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:48.229+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:49.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:49.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:49.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:49.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:50.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:50.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:50.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:50.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:51.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:51.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:51.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:51.217+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:52.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:52.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:52.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:52.241+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:53.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:53.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:53.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:53.273+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:54.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:54.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:54.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:54.249+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:55.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:55.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:55.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:55.221+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:56.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:56.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:56.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:56.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:57.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:57.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:57.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:57.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:58.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:58.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:58.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:58.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:59.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:52:59.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:52:59.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:52:59.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:00.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:00.001+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:00.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:00.221+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:00.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:00.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:01.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:01.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:02.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:02.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:02.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:02.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:03.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:03.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:03.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:03.093+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:04.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:04.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:04.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:04.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:05.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:05.069+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:05.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:05.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:06.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:06.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:06.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:06.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:07.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:07.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:07.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:07.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:08.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:08.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:08.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:08.169+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:09.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:09.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:09.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:09.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:10.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:10.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:10.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:10.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:11.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:11.165+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:11.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:11.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:12.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:12.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:12.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:12.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:13.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:13.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:13.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:13.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:14.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:14.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:14.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:14.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:15.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:15.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:15.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:15.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:16.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:16.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:16.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:16.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:17.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:17.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:17.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:17.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:18.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:18.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:18.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:18.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:19.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:19.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:19.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:19.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:20.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:20.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:20.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:20.165+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:21.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:21.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:21.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:21.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:22.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:22.160+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:22.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:22.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:23.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:23.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:23.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:23.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:24.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:24.076+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:24.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:24.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:25.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:25.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:25.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:25.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:26.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:26.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:26.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:26.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:27.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:27.108+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:27.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:27.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:28.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:28.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:28.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:28.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:29.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:29.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:29.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:29.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:30.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:30.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:30.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:30.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:31.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:31.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:31.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:31.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:32.099 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:32.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:32.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:32.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:33.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:33.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:33.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:33.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:34.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:34.064+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:34.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:34.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:35.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:35.048+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:35.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:35.160+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:36.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:36.076+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:36.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:36.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:37.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:37.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:37.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:37.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:38.134 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:38.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:38.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:38.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:39.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:39.092+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:39.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:39.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:40.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:40.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:40.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:40.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:41.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:41.148+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:41.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:41.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:42.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:42.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:42.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:42.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:43.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:43.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:43.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:43.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:44.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:44.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:44.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:44.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:45.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:45.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:45.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:45.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:46.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:46.148+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:46.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:46.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:47.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:47.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:47.358 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:47.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:48.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:48.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:48.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:48.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:49.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:49.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:49.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:49.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:50.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:50.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:50.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:50.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:51.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:51.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:51.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:51.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:52.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:52.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:52.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:52.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:53.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:53.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:53.351 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:53.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:54.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:54.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:54.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:54.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:55.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:55.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:55.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:55.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:56.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:56.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:56.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:56.376+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:57.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:57.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:57.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:57.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:58.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:58.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:58.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:58.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:59.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:53:59.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:53:59.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:53:59.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:00.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:00.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:00.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:00.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:01.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:01.192+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:01.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:01.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:02.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:02.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:02.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:02.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:03.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:03.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:03.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:03.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:04.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:04.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:04.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:04.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:05.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:05.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:05.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:05.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:06.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:06.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:06.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:06.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:07.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:07.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:07.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:07.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:08.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:08.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:08.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:08.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:09.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:09.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:09.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:09.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:10.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:10.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:10.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:10.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:11.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:11.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:11.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:11.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:12.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:12.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:12.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:12.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:13.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:13.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:13.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:13.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:14.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:14.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:14.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:14.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:15.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:15.167+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:15.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:15.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:16.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:16.211+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:16.429 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:16.427+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:17.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:17.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:17.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:17.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:18.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:18.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:18.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:18.439+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:19.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:19.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:19.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:19.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:20.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:20.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:20.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:20.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:21.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:21.167+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:21.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:21.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:22.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:22.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:22.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:22.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:23.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:23.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:23.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:23.347+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:24.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:24.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:24.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:24.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:25.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:25.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:25.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:25.347+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:26.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:26.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:26.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:26.371+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:27.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:27.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:27.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:27.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:28.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:28.135+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:28.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:28.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:29.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:29.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:29.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:29.359+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:30.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:30.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:30.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:30.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:31.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:31.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:31.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:31.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:32.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:32.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:32.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:32.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:33.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:33.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:33.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:33.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:34.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:34.211+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:34.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:34.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:35.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:35.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:35.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:35.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:36.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:36.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:36.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:36.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:37.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:37.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:37.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:37.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:38.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:38.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:38.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:38.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:39.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:39.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:39.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:39.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:40.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:40.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:40.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:40.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:41.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:41.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:41.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:41.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:42.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:42.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:42.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:42.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:43.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:43.179+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:43.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:43.211+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:44.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:44.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:44.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:44.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:45.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:45.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:45.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:45.267+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:46.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:46.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:46.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:46.311+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:47.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:47.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:47.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:47.355+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:48.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:48.183+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:48.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:48.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:49.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:49.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:49.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:49.291+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:50.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:50.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:50.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:50.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:51.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:51.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:51.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:51.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:52.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:52.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:52.303 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:52.299+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:53.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:53.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:53.303 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:53.299+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:54.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:54.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:54.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:54.311+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:55.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:55.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:55.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:55.311+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:56.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:56.303+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:56.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:56.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:57.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:57.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:57.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:57.371+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:58.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:58.347+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:58.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:58.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:59.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:54:59.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:54:59.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:54:59.423+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:00.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:00.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:00.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:00.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:01.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:01.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:01.429 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:01.427+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:02.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:02.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:02.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:02.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:03.422 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:03.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:03.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:03.431+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:04.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:04.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:04.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:04.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:05.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:05.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:05.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:05.475+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:06.386 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:06.383+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:06.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:06.459+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:07.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:07.414+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:07.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:07.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:08.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:08.366+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:08.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:08.458+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:09.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:09.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:09.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:09.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:10.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:10.402+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:10.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:10.454+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:11.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:11.422+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:11.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:11.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:12.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:12.390+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:12.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:12.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:13.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:13.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:13.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:13.438+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:14.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:14.414+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:14.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:14.478+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:15.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:15.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:15.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:15.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:16.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:16.482+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:16.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:16.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:17.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:17.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:17.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:17.530+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:18.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:18.514+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:18.555 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:18.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:19.544 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:19.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:19.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:19.546+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:20.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:20.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:20.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:20.598+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:21.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:21.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:21.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:21.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:22.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:22.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:22.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:22.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:23.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:23.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:23.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:23.598+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:24.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:24.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:24.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:24.550+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:25.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:25.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:25.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:25.534+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:26.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:26.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:26.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:26.494+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:27.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:27.518+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:27.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:27.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:28.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:28.482+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:28.573 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:28.570+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:29.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:29.442+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:29.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:29.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:30.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:30.406+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:30.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:30.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:31.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:31.410+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:31.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:31.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:32.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:32.402+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:32.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:32.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:33.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:33.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:33.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:33.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:34.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:34.382+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:34.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:34.414+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:35.381 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:35.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:35.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:35.402+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:36.373 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:36.370+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:36.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:36.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:37.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:37.358+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:37.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:37.366+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:38.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:38.338+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:38.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:38.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:39.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:39.362+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:39.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:39.386+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:40.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:40.346+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:40.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:40.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:41.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:41.354+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:41.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:41.434+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:42.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:42.394+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:42.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:42.414+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:43.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:43.410+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:43.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:43.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:44.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:44.394+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:44.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:44.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:45.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:45.402+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:45.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:45.434+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:46.386 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:46.382+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:46.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:46.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:47.371 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:47.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:47.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:47.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:48.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:48.358+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:48.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:48.382+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:49.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:49.322+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:49.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:49.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:50.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:50.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:50.405 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:50.402+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:51.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:51.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:51.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:51.438+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:52.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:52.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:52.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:52.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:53.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:53.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:53.418 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:53.414+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:54.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:54.334+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:54.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:54.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:55.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:55.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:55.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:55.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:56.358 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:56.354+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:56.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:56.414+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:57.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:57.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:57.403 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:57.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:58.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:58.318+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:58.402 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:58.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:59.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:55:59.318+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:55:59.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:55:59.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:00.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:00.349+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:00.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:00.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:01.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:01.377+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:01.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:01.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:02.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:02.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:02.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:02.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:03.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:03.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:03.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:03.417+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:04.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:04.397+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:04.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:04.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:05.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:05.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:05.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:05.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:06.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:06.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:06.446 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:06.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:07.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:07.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:07.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:07.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:08.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:08.373+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:08.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:08.421+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:09.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:09.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:09.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:09.409+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:10.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:10.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:10.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:10.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:11.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:11.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:11.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:11.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:12.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:12.433+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:12.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:12.465+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:13.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:13.401+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:13.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:13.509+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:14.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:14.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:14.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:14.465+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:15.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:15.401+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:15.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:15.501+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:16.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:16.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:16.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:16.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:17.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:17.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:17.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:17.509+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:18.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:18.417+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:18.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:18.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:19.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:19.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:19.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:19.529+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:20.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:20.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:20.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:20.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:21.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:21.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:21.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:21.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:22.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:22.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:22.485 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:22.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:23.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:23.377+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:23.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:23.521+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:24.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:24.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:24.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:24.557+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:25.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:25.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:25.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:25.609+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:26.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:26.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:26.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:26.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:27.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:27.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:27.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:27.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:28.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:28.517+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:28.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:28.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:29.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:29.533+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:29.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:29.609+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:30.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:30.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:30.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:30.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:31.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:31.553+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:31.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:31.569+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:32.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:32.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:32.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:32.529+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:33.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:33.525+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:33.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:33.549+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:34.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:34.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:34.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:34.577+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:35.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:35.541+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:35.569 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:35.565+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:36.541 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:36.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:36.603 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:36.597+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:37.555 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:37.549+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:37.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:37.569+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:38.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:38.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:38.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:38.601+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:39.627 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:39.621+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:39.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:39.637+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:40.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:40.597+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:40.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:40.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:41.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:41.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:41.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:41.701+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:42.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:42.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:42.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:42.701+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:43.606 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:43.601+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:43.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:43.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:44.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:44.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:44.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:44.693+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:45.619 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:45.613+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:45.745 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:45.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:46.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:46.573+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:46.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:46.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:47.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:47.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:47.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:47.689+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:48.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:48.605+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:48.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:48.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:49.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:49.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:49.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:49.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:50.618 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:50.613+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:50.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:50.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:51.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:51.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:51.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:51.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:52.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:52.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:52.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:52.864+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:53.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:53.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:53.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:53.884+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:54.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:54.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:54.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:54.852+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:55.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:55.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:55.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:55.864+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:56.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:56.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:56.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:56.908+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:57.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:57.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:57.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:57.892+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:58.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:58.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:58.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:58.864+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:59.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:56:59.820+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:56:59.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:56:59.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:00.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:00.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:00.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:00.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:01.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:01.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:01.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:01.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:02.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:02.716+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:02.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:02.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:03.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:03.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:03.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:03.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:04.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:04.744+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:04.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:04.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:05.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:05.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:05.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:05.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:06.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:06.712+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:06.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:06.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:07.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:07.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:07.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:07.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:08.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:08.768+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:08.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:08.984+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:09.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:09.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:09.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:09.984+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:10.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:10.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:10.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:10.984+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:11.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:11.764+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:12.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:12.028+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:12.745 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:12.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:13.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:13.008+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:13.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:13.700+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:14.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:14.036+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:14.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:14.736+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:15.010 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:15.004+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:15.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:15.784+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:16.023 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:16.016+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:16.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:16.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:16.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:16.980+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:17.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:17.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:17.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:17.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:18.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:18.696+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:18.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:18.980+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:19.702 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:19.696+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:19.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:19.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:20.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:20.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:20.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:20.972+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:21.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:21.692+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:21.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:21.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:22.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:22.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:22.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:22.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:23.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:23.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:24.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:24.000+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:24.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:24.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:25.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:25.012+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:25.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:25.744+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:25.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:25.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:26.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:26.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:26.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:26.952+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:27.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:27.756+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:27.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:27.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:28.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:28.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:28.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:28.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:29.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:29.772+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:29.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:29.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:30.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:30.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:30.905 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:30.904+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:31.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:31.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:31.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:31.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:32.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:32.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:32.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:32.872+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:33.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:33.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:33.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:33.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:34.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:34.896+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:34.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:34.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:35.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:35.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:35.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:35.892+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:36.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:36.876+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:36.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:36.884+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:37.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:37.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:37.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:37.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:38.874 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:38.872+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:38.967 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:38.964+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:39.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:39.868+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:39.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:39.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:40.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:40.920+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:40.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:40.940+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:41.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:41.932+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:41.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:41.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:42.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:42.896+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:42.943 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:42.940+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:43.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:43.884+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:43.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:43.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:44.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:44.896+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:45.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:45.012+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:45.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:45.879+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:46.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:46.007+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:46.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:46.887+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:47.053 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:47.051+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:47.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:47.855+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:48.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:48.075+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:48.815 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:48.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:49.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:49.087+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:49.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:49.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:50.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:50.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:50.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:50.835+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:51.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:51.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:51.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:51.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:52.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:52.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:52.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:52.839+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:53.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:53.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:53.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:53.851+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:54.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:54.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:54.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:54.859+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:55.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:55.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:55.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:55.843+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:56.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:56.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:56.895 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:56.891+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:57.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:57.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:57.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:57.895+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:58.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:58.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:58.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:58.911+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:59.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:57:59.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:57:59.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:57:59.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:00.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:00.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:00.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:00.927+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:01.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:01.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:01.933 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:01.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:02.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:02.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:02.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:02.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:03.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:03.291+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:03.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:03.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:04.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:04.335+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:05.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:05.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:05.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:05.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:06.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:06.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:06.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:06.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:07.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:07.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:07.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:07.351+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:08.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:08.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:08.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:08.307+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:09.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:09.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:09.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:09.279+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:10.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:10.047+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:10.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:10.311+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:11.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:11.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:11.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:11.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:12.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:12.063+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:12.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:12.347+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:13.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:13.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:13.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:13.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:14.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:14.031+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:14.414 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:14.411+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:15.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:15.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:15.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:15.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:16.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:15.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:16.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:16.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:17.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:16.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:17.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:17.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:18.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:18.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:18.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:18.363+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:19.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:19.031+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:19.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:19.367+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:20.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:20.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:20.385 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:20.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:21.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:21.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:21.369 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:21.367+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:22.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:22.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:22.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:22.347+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:23.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:23.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:23.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:23.331+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:24.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:24.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:24.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:24.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:25.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:25.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:25.274 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:25.271+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:26.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:26.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:26.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:26.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:27.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:27.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:27.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:27.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:28.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:28.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:28.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:28.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:29.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:29.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:29.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:29.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:30.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:30.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:30.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:30.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:31.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:31.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:31.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:31.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:32.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:32.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:32.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:32.171+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:33.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:33.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:33.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:33.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:34.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:34.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:34.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:34.135+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:35.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:35.127+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:35.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:35.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:36.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:36.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:36.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:36.107+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:37.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:37.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:37.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:37.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:38.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:38.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:38.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:38.075+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:39.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:39.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:39.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:39.035+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:39.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:39.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:40.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:40.047+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:41.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:41.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:41.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:41.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:41.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:41.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:42.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:42.042+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:43.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:43.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:43.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:43.086+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:44.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:44.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:44.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:44.074+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:45.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:45.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:45.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:45.118+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:46.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:46.078+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:46.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:46.094+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:47.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:47.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:47.132 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:47.130+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:48.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:48.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:48.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:48.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:49.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:49.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:49.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:49.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:50.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:50.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:50.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:50.110+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:51.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:51.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:51.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:51.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:52.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:52.110+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:52.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:52.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:53.099 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:53.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:53.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:53.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:54.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:54.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:54.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:54.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:55.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:55.126+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:55.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:55.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:56.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:56.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:56.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:56.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:57.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:57.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:57.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:57.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:58.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:58.078+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:58.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:58.222+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:59.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:58:59.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:58:59.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:58:59.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:00.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:00.078+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:00.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:00.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:01.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:01.118+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:01.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:01.322+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:02.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:02.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:02.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:02.354+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:03.101 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:03.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:03.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:03.394+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:04.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:04.122+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:04.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:04.434+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:05.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:05.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:05.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:05.402+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:06.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:06.122+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:06.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:06.418+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:07.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:07.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:07.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:07.370+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:08.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:08.102+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:08.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:08.382+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:09.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:09.070+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:09.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:09.386+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:10.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:10.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:10.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:10.346+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:11.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:11.026+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:11.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:11.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:12.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:12.074+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:12.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:12.322+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:13.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:13.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:13.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:13.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:14.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:14.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:14.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:14.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:15.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:15.106+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:15.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:15.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:16.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:16.130+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:16.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:16.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:17.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:17.162+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:17.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:17.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:18.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:18.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:18.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:18.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:19.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:19.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:19.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:19.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:20.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:20.230+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:20.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:20.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:21.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:21.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:21.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:21.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:22.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:22.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:22.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:22.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:23.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:23.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:23.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:23.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:24.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:24.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:24.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:24.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:25.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:25.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:25.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:25.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:26.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:26.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:26.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:26.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:27.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:27.174+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:27.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:27.302+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:28.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:28.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:28.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:28.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:29.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:29.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:29.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:29.274+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:30.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:30.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:30.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:30.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:31.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:31.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:31.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:31.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:32.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:32.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:32.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:32.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:33.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:33.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:33.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:33.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:34.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:34.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:34.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:34.326+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:35.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:35.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:35.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:35.297+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:36.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:36.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:36.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:36.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:37.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:37.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:37.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:37.321+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:38.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:38.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:38.337 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:38.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:39.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:39.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:39.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:39.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:40.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:40.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:40.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:40.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:41.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:41.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:41.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:41.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:42.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:42.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:42.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:42.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:43.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:43.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:43.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:43.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:44.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:44.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:44.405 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:44.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:45.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:45.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:45.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:45.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:46.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:46.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:46.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:46.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:47.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:47.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:47.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:47.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:48.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:48.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:48.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:48.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:49.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:49.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:49.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:49.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:50.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:50.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:50.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:50.369+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:51.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:51.381+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:51.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:51.405+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:52.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:52.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:52.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:52.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:53.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:53.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:53.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:53.401+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:54.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:54.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:54.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:54.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:55.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:55.421+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:55.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:55.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:56.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:56.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:56.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:56.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:57.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:57.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:57.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:57.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:58.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:58.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:58.398 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:58.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:59.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T22:59:59.393+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T22:59:59.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T22:59:59.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:00.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:00.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:00.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:00.417+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:01.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:01.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:01.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:01.445+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:02.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:02.421+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:02.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:02.469+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:03.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:03.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:03.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:03.445+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:04.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:04.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:04.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:04.413+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:05.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:05.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:05.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:05.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:06.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:06.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:06.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:06.497+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:07.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:07.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:07.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:07.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:08.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:08.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:08.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:08.489+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:09.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:09.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:09.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:09.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:10.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:10.433+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:10.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:10.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:11.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:11.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:11.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:11.497+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:12.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:12.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:12.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:12.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:13.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:13.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:13.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:13.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:14.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:14.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:14.470 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:14.465+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:15.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:15.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:15.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:15.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:16.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:16.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:16.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:16.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:17.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:17.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:17.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:17.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:18.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:18.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:18.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:18.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:19.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:19.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:19.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:19.493+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:20.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:20.473+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:20.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:20.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:21.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:21.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:21.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:21.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:22.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:22.453+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:22.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:22.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:23.455 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:23.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:23.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:23.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:24.456 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:24.453+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:24.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:24.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:25.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:25.405+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:25.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:25.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:26.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:26.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:26.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:26.421+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:27.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:27.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:27.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:27.405+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:28.422 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:28.417+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:28.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:28.437+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:29.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:29.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:29.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:29.469+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:30.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:30.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:30.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:30.460+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:31.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:31.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:31.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:31.492+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:32.449 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:32.444+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:32.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:32.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:33.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:33.464+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:33.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:33.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:34.485 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:34.480+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:34.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:34.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:35.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:35.512+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:35.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:35.532+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:36.506 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:36.500+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:36.588 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:36.584+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:37.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:37.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:37.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:37.552+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:38.550 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:38.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:38.571 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:38.564+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:39.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:39.520+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:39.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:39.572+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:40.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:40.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:40.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:40.556+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:41.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:41.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:41.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:41.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:42.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:42.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:42.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:42.596+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:43.456 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:43.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:43.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:43.588+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:44.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:44.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:44.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:44.616+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:45.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:45.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:45.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:45.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:46.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:46.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:46.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:46.612+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:47.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:47.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:47.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:47.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:48.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:48.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:48.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:48.536+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:49.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:49.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:49.516 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:49.512+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:50.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:50.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:50.554 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:50.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:51.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:51.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:51.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:51.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:52.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:52.464+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:52.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:52.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:53.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:53.480+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:53.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:53.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:54.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:54.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:54.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:54.508+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:55.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:55.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:55.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:55.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:56.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:56.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:56.550 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:56.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:57.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:57.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:57.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:57.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:58.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:58.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:58.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:58.532+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:59.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:00:59.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:00:59.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:00:59.552+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:00.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:00.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:00.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:00.596+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:01.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:01.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:01.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:01.568+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:02.499 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:02.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:02.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:02.572+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:03.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:03.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:03.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:03.572+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:04.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:04.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:04.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:04.588+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:05.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:05.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:05.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:05.636+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:06.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:06.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:06.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:06.620+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:07.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:07.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:07.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:07.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:08.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:08.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:08.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:08.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:09.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:09.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:09.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:09.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:10.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:10.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:10.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:10.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:11.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:11.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:11.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:11.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:12.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:12.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:12.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:12.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:13.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:13.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:13.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:13.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:14.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:14.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:14.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:14.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:15.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:15.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:15.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:15.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:16.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:16.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:16.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:16.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:17.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:17.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:17.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:17.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:18.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:18.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:18.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:18.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:19.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:19.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:19.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:19.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:20.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:20.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:20.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:20.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:21.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:21.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:21.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:21.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:22.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:22.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:22.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:22.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:23.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:23.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:23.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:23.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:24.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:24.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:24.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:24.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:25.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:25.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:25.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:25.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:26.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:26.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:26.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:26.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:27.358 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:27.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:27.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:27.795+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:28.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:28.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:28.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:28.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:29.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:29.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:29.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:29.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:30.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:30.371+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:30.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:30.795+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:31.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:31.335+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:31.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:31.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:32.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:32.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:32.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:32.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:33.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:33.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:33.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:33.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:34.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:34.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:34.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:34.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:35.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:35.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:35.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:35.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:36.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:36.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:36.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:36.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:37.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:37.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:37.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:37.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:38.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:38.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:38.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:38.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:39.330 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:39.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:39.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:39.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:40.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:40.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:40.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:40.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:41.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:41.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:41.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:41.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:42.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:42.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:42.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:42.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:43.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:43.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:43.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:43.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:44.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:44.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:44.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:44.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:45.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:45.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:45.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:45.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:46.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:46.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:46.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:46.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:47.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:47.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:47.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:47.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:48.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:48.195+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:48.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:48.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:49.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:49.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:49.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:49.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:50.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:50.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:50.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:50.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:51.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:51.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:51.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:51.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:52.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:52.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:52.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:52.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:53.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:53.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:53.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:53.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:54.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:54.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:54.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:54.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:55.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:55.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:55.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:55.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:56.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:56.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:56.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:56.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:57.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:57.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:57.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:57.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:58.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:58.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:58.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:58.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:59.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:01:59.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:01:59.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:01:59.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:00.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:00.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:00.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:00.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:01.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:01.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:01.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:01.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:02.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:02.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:02.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:02.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:03.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:03.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:03.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:03.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:04.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:04.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:04.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:04.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:05.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:05.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:05.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:05.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:06.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:06.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:06.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:06.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:07.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:07.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:07.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:07.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:08.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:08.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:08.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:08.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:09.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:09.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:09.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:09.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:10.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:10.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:10.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:10.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:11.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:11.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:11.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:11.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:12.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:12.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:12.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:12.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:13.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:13.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:13.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:13.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:14.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:14.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:14.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:14.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:15.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:15.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:15.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:15.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:16.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:16.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:16.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:16.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:17.337 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:17.334+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:17.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:17.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:18.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:18.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:18.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:18.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:19.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:19.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:19.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:19.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:20.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:20.354+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:20.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:20.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:21.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:21.350+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:21.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:21.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:22.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:22.362+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:22.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:22.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:23.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:23.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:23.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:23.782+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:24.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:24.390+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:24.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:24.830+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:25.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:25.362+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:25.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:25.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:26.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:26.414+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:26.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:26.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:27.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:27.442+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:27.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:27.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:28.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:28.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:28.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:28.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:29.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:29.462+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:29.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:29.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:30.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:30.506+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:30.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:30.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:31.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:31.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:31.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:31.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:32.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:32.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:32.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:32.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:33.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:33.542+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:33.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:33.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:34.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:34.522+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:34.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:34.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:35.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:35.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:35.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:35.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:36.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:36.534+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:36.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:36.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:37.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:37.538+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:37.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:37.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:38.569 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:38.566+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:38.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:38.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:39.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:39.582+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:39.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:39.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:40.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:40.562+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:40.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:40.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:41.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:41.594+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:41.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:41.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:42.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:42.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:42.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:42.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:43.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:43.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:43.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:43.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:44.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:44.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:44.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:44.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:45.580 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:45.578+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:45.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:45.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:46.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:46.558+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:46.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:46.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:47.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:47.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:47.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:47.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:48.535 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:48.530+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:48.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:48.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:49.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:49.538+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:49.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:49.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:50.564 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:50.562+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:50.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:50.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:51.553 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:51.550+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:51.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:51.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:52.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:52.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:52.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:52.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:53.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:53.562+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:53.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:53.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:54.521 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:54.518+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:54.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:54.722+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:55.563 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:55.558+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:55.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:55.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:56.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:56.530+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:56.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:56.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:57.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:57.506+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:57.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:57.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:58.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:58.502+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:58.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:58.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:59.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:02:59.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:02:59.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:02:59.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:00.578 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:00.574+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:00.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:00.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:01.623 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:01.618+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:01.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:01.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:02.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:02.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:02.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:02.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:03.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:03.682+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:03.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:03.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:04.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:04.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:04.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:04.802+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:05.641 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:05.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:05.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:05.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:06.624 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:06.622+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:06.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:06.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:07.663 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:07.658+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:07.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:07.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:08.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:08.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:08.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:08.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:09.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:09.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:09.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:09.806+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:10.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:10.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:10.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:10.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:11.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:11.753+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:11.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:11.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:12.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:12.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:12.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:12.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:13.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:13.685+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:13.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:13.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:14.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:14.709+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:14.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:14.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:15.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:15.681+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:15.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:15.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:16.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:16.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:16.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:16.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:17.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:17.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:17.943 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:17.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:18.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:18.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:18.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:18.905+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:19.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:19.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:19.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:19.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:20.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:20.681+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:20.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:20.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:21.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:21.733+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:21.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:21.853+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:22.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:22.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:22.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:22.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:23.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:23.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:23.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:23.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:24.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:24.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:24.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:24.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:25.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:25.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:25.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:25.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:26.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:26.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:26.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:26.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:27.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:27.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:27.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:27.881+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:28.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:28.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:28.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:28.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:29.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:29.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:29.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:29.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:30.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:30.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:30.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:30.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:31.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:31.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:31.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:31.945+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:32.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:32.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:32.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:32.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:33.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:33.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:33.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:33.945+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:34.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:34.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:34.917 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:34.913+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:35.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:35.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:35.941 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:35.937+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:36.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:36.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:36.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:36.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:37.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:37.901+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:37.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:37.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:38.905 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:38.901+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:38.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:38.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:39.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:39.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:39.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:39.945+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:40.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:40.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:40.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:40.929+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:41.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:41.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:41.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:41.973+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:43.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:42.997+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:43.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:43.021+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:43.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:43.985+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:44.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:44.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:44.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:44.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:45.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:45.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:46.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:46.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:46.059 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:46.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:46.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:46.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:47.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:47.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:48.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:48.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:48.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:48.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:49.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:49.001+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:49.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:49.021+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:50.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:50.029+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:50.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:50.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:51.043 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:51.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:51.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:51.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:52.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:52.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:52.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:52.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:53.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:53.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:53.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:53.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:54.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:54.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:54.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:54.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:55.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:55.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:55.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:55.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:56.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:56.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:56.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:56.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:57.018 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:57.013+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:57.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:57.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:58.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:58.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:58.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:58.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:59.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:03:59.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:03:59.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:03:59.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:00.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:00.081+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:00.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:00.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:01.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:01.053+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:01.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:01.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:02.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:02.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:02.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:02.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:03.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:03.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:03.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:03.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:04.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:04.052+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:04.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:04.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:05.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:05.080+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:05.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:05.088+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:06.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:06.076+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:06.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:06.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:07.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:07.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:07.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:07.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:08.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:08.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:08.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:08.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:09.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:09.148+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:09.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:09.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:10.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:10.196+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:10.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:10.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:11.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:11.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:11.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:11.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:12.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:12.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:12.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:12.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:13.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:13.148+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:13.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:13.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:14.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:14.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:14.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:14.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:15.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:15.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:15.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:15.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:16.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:16.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:16.267 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:16.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:17.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:17.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:17.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:17.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:18.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:18.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:18.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:18.320+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:19.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:19.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:19.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:19.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:20.196 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:20.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:20.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:20.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:21.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:21.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:21.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:21.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:22.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:22.224+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:22.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:22.340+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:23.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:23.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:23.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:23.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:24.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:24.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:24.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:24.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:25.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:25.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:25.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:25.376+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:26.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:26.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:26.418 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:26.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:27.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:27.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:27.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:27.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:28.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:28.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:28.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:28.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:29.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:29.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:29.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:29.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:30.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:30.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:30.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:30.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:31.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:31.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:31.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:31.424+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:32.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:32.164+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:32.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:32.456+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:33.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:33.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:33.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:33.464+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:34.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:34.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:34.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:34.472+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:35.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:35.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:35.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:35.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:36.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:36.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:36.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:36.472+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:37.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:37.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:37.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:37.468+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:38.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:38.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:38.458 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:38.452+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:39.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:39.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:39.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:39.428+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:40.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:40.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:40.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:40.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:41.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:41.208+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:41.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:41.444+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:42.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:42.196+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:42.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:42.456+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:43.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:43.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:43.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:43.428+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:44.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:44.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:44.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:44.396+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:45.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:45.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:45.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:45.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:46.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:46.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:46.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:46.384+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:47.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:47.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:47.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:47.356+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:48.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:48.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:48.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:48.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:49.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:49.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:49.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:49.320+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:50.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:50.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:50.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:50.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:51.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:51.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:51.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:51.316+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:52.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:52.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:52.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:52.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:53.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:53.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:53.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:53.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:54.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:54.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:54.338 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:54.336+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:55.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:55.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:55.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:55.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:56.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:56.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:56.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:56.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:57.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:57.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:57.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:57.379+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:58.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:58.343+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:58.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:58.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:59.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:04:59.359+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:04:59.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:04:59.367+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:00.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:00.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:00.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:00.379+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:01.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:01.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:01.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:01.415+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:02.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:02.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:02.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:02.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:03.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:03.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:03.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:03.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:04.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:04.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:04.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:04.459+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:05.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:05.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:05.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:05.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:06.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:06.411+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:06.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:06.531+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:07.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:07.371+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:07.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:07.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:08.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:08.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:08.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:08.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:09.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:09.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:09.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:09.515+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:10.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:10.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:10.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:10.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:10.739 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T23:05:10.735+0000 7fd5e8fc6700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x560328be4050 msgr2=0x7fd5cc0d1930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T23:05:10.802 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T23:05:10.799+0000 7fd5e97c7700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6804/20057,v1:172.21.15.178:6806/20057] conn(0x560328be4050 msgr2=0x7fd5cc0d1910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T23:05:11.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:11.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:11.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:11.499+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:12.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:12.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:12.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:12.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:13.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:13.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:13.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:13.527+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:14.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:14.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:14.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:14.555+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:15.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:15.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:15.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:15.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:16.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:16.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:16.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:16.579+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:17.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:17.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:17.592 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:17.591+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:18.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:18.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:18.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:18.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:19.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:19.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:19.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:19.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:20.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:20.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:20.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:20.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:21.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:21.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:21.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:21.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:22.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:22.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:22.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:22.619+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:23.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:23.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:23.592 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:23.591+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:24.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:24.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:24.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:24.567+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:25.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:25.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:25.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:25.531+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:26.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:26.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:26.538 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:26.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:27.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:27.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:27.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:27.527+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:28.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:28.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:28.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:28.567+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:29.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:29.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:29.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:29.543+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:30.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:30.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:30.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:30.583+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:31.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:31.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:31.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:31.571+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:32.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:32.183+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:32.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:32.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:33.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:33.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:33.573 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:33.571+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:34.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:34.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:34.587 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:34.583+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:35.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:35.179+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:35.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:35.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:36.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:36.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:36.618 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:36.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:37.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:37.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:37.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:37.611+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:38.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:38.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:38.593 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:38.591+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:39.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:39.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:39.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:39.635+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:40.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:40.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:40.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:40.595+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:41.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:41.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:41.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:41.627+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:42.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:42.183+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:42.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:42.587+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:43.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:43.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:43.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:43.587+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:44.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:44.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:44.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:44.619+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:45.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:45.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:45.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:45.611+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:46.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:46.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:46.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:46.631+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:47.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:47.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:47.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:47.671+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:48.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:48.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:48.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:48.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:49.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:49.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:49.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:49.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:50.196 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:50.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:50.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:50.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:51.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:51.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:51.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:51.638+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:52.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:52.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:52.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:52.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:53.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:53.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:53.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:53.670+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:54.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:54.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:54.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:54.682+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:55.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:55.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:55.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:55.706+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:56.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:56.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:56.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:56.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:57.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:57.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:57.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:57.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:58.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:58.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:58.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:58.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:59.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:05:59.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:05:59.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:05:59.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:00.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:00.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:00.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:00.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:01.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:01.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:01.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:01.654+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:02.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:02.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:02.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:02.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:03.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:03.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:03.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:03.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:04.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:04.222+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:04.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:04.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:05.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:05.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:05.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:05.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:06.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:06.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:06.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:06.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:07.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:07.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:07.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:07.710+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:08.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:08.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:08.718 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:08.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:09.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:09.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:09.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:09.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:10.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:10.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:10.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:10.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:11.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:11.326+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:11.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:11.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:12.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:12.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:12.682 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:12.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:13.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:13.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:13.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:13.682+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:14.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:14.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:14.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:14.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:15.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:15.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:15.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:15.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:16.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:16.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:16.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:16.754+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:17.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:17.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:17.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:17.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:18.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:18.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:18.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:18.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:19.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:19.282+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:19.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:19.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:20.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:20.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:20.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:20.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:21.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:21.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:21.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:21.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:22.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:22.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:22.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:22.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:23.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:23.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:23.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:23.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:24.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:24.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:24.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:24.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:25.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:25.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:25.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:25.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:26.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:26.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:26.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:26.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:27.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:27.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:27.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:27.774+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:28.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:28.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:28.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:28.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:29.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:29.202+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:29.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:29.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:30.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:30.234+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:30.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:30.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:31.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:31.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:31.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:31.722+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:32.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:32.186+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:32.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:32.674+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:33.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:33.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:33.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:33.662+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:34.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:34.174+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:34.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:34.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:35.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:35.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:35.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:35.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:36.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:36.186+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:36.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:36.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:37.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:37.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:37.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:37.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:38.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:38.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:38.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:38.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:39.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:39.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:39.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:39.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:40.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:40.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:40.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:40.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:41.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:41.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:41.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:41.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:42.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:42.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:42.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:42.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:43.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:43.174+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:43.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:43.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:44.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:44.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:44.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:44.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:45.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:45.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:45.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:45.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:46.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:46.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:46.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:46.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:47.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:47.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:47.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:47.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:48.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:48.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:48.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:48.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:49.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:49.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:49.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:49.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:50.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:50.065+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:50.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:50.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:51.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:51.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:51.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:51.933+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:52.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:52.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:52.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:52.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:53.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:53.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:53.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:53.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:54.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:54.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:54.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:54.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:55.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:55.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:55.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:55.905+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:56.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:56.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:56.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:56.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:57.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:57.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:57.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:57.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:58.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:58.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:58.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:58.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:59.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:06:59.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:06:59.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:06:59.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:00.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:00.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:00.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:00.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:01.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:01.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:01.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:01.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:02.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:02.021+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:02.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:02.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:03.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:03.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:03.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:03.925+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:04.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:04.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:04.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:04.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:05.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:05.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:05.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:05.909+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:06.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:06.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:06.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:06.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:07.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:07.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:07.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:07.889+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:08.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:08.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:08.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:08.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:09.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:09.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:09.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:09.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:10.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:10.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:10.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:10.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:11.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:11.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:11.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:11.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:12.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:12.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:12.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:12.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:13.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:13.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:13.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:13.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:14.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:14.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:14.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:14.749+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:15.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:15.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:15.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:15.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:16.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:16.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:16.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:16.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:17.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:17.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:17.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:17.657+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:18.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:18.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:18.712 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:18.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:19.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:19.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:19.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:19.669+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:20.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:20.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:20.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:20.681+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:21.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:21.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:21.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:21.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:22.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:22.169+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:22.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:22.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:23.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:23.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:23.627 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:23.621+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:24.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:24.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:24.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:24.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:25.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:25.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:25.694 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:25.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:26.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:26.185+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:26.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:26.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:27.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:27.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:27.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:27.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:28.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:28.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:28.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:28.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:29.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:29.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:29.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:29.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:30.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:30.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:30.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:30.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:31.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:31.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:31.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:31.849+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:32.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:32.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:32.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:32.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:33.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:33.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:33.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:33.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:34.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:34.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:34.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:34.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:35.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:35.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:35.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:35.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:36.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:36.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:36.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:36.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:37.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:37.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:37.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:37.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:38.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:38.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:38.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:38.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:39.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:39.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:39.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:39.904+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:40.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:40.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:40.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:40.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:41.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:41.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:41.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:41.952+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:42.386 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:42.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:42.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:42.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:43.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:43.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:43.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:43.940+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:44.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:44.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:44.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:44.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:45.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:45.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:45.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:45.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:46.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:46.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:46.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:46.972+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:47.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:47.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:47.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:47.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:48.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:48.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:48.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:48.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:49.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:49.220+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:49.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:49.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:50.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:50.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:50.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:50.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:51.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:51.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:51.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:51.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:52.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:52.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:52.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:52.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:53.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:53.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:53.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:53.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:54.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:54.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:54.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:54.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:55.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:55.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:55.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:55.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:56.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:56.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:56.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:56.896+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:57.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:57.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:57.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:57.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:58.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:58.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:58.907 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:58.900+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:59.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:07:59.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:07:59.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:07:59.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:00.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:00.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:00.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:00.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:01.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:01.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:01.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:01.952+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:02.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:02.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:02.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:02.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:03.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:03.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:03.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:03.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:04.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:04.156+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:04.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:04.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:05.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:05.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:05.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:05.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:06.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:06.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:06.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:06.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:07.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:07.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:07.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:07.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:08.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:08.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:08.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:08.896+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:09.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:09.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:09.905 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:09.900+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:10.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:10.100+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:10.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:10.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:11.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:11.060+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:11.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:11.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:12.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:12.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:12.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:12.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:13.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:13.080+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:13.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:13.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:14.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:14.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:14.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:14.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:15.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:15.088+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:15.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:15.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:16.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:16.100+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:16.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:16.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:17.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:17.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:17.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:17.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:18.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:18.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:18.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:18.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:19.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:19.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:19.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:19.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:20.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:20.180+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:20.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:20.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:21.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:21.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:21.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:21.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:22.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:22.164+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:22.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:22.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:23.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:23.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:23.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:23.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:24.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:24.220+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:24.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:24.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:25.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:25.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:25.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:25.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:26.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:26.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:26.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:26.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:27.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:27.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:27.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:27.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:28.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:28.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:28.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:28.836+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:29.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:29.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:29.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:29.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:30.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:30.200+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:30.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:30.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:31.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:31.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:31.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:31.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:32.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:32.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:32.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:32.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:33.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:33.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:34.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:34.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:34.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:34.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:35.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:35.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:35.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:35.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:36.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:36.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:36.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:36.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:36.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:36.979+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:37.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:37.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:38.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:38.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:38.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:38.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:39.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:39.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:39.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:39.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:40.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:40.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:40.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:40.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:41.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:41.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:41.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:41.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:42.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:42.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:42.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:42.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:43.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:43.035+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:43.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:43.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:44.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:44.007+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:44.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:44.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:45.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:45.035+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:45.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:45.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:46.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:46.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:46.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:46.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:46.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:46.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:47.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:47.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:48.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:48.031+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:48.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:48.195+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:49.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:49.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:49.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:49.223+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:50.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:50.031+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:50.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:50.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:51.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:51.007+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:51.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:51.195+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:51.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:51.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:52.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:52.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:53.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:53.007+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:53.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:53.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:54.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:54.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:54.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:54.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:55.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:55.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:55.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:55.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:56.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:56.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:56.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:56.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:57.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:57.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:57.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:57.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:58.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:58.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:58.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:58.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:59.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:08:59.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:08:59.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:08:59.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:00.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:00.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:00.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:00.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:01.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:01.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:01.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:01.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:02.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:02.035+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:02.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:02.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:03.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:03.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:03.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:03.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:04.101 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:04.099+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:04.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:04.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:05.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:05.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:05.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:05.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:06.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:06.087+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:06.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:06.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:07.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:07.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:07.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:07.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:08.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:08.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:08.351 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:08.347+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:09.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:09.087+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:09.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:09.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:10.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:10.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:10.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:10.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:11.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:11.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:11.330 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:11.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:12.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:12.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:12.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:12.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:13.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:13.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:13.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:13.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:14.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:14.027+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:14.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:14.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:14.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:14.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:15.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:15.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:15.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:15.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:16.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:16.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:16.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:16.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:17.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:17.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:17.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:17.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:18.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:18.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:18.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:18.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:19.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:19.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:20.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:20.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:20.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:20.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:21.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:21.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:21.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:21.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:22.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:22.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:22.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:22.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:23.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:23.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:23.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:23.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:24.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:24.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:24.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:24.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:25.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:25.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:25.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:25.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:26.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:26.055+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:26.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:26.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:27.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:27.027+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:27.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:27.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:28.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:28.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:28.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:28.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:28.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:28.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:29.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:29.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:30.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:30.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:30.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:30.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:31.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:31.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:31.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:31.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:31.893 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T23:09:31.890+0000 7fd5e8fc6700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560328bd4930 msgr2=0x560328c74920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T23:09:31.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:31.970+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:32.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:32.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:32.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:32.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:33.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:33.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:33.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:33.970+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:34.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:34.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:34.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:34.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:35.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:35.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:35.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:35.954+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:36.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:36.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:36.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:36.958+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:37.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:37.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:37.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:37.934+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:38.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:38.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:38.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:38.966+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:39.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:39.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:40.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:40.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:40.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:40.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:40.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:40.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:41.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:41.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:41.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:41.966+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:42.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:42.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:42.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:42.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:43.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:43.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:43.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:43.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:44.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:44.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:44.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:44.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:45.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:45.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:45.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:45.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:46.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:46.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:46.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:46.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:47.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:47.094+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:47.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:47.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:48.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:48.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:48.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:48.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:49.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:49.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:49.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:49.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:50.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:50.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:50.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:50.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:51.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:51.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:51.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:51.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:52.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:52.058+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:52.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:52.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:53.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:53.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:53.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:53.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:54.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:54.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:54.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:54.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:55.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:55.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:55.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:55.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:56.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:56.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:56.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:56.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:56.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:56.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:57.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:57.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:57.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:57.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:58.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:58.934+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:59.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:09:59.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:09:59.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:09:59.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:00.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:00.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:00.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:00.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:01.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:01.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:01.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:01.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:02.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:02.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:02.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:02.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:03.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:03.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:03.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:03.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:04.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:04.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:04.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:04.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:05.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:05.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:05.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:05.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:06.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:06.058+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:06.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:06.918+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:07.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:07.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:07.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:07.954+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:08.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:08.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:08.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:08.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:09.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:09.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:09.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:09.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:10.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:10.058+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:10.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:10.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:11.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:11.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:11.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:11.934+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:12.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:12.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:12.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:12.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:13.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:13.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:13.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:13.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:14.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:14.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:14.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:14.914+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:15.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:15.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:15.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:15.942+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:16.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:16.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:16.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:16.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:17.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:17.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:17.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:17.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:18.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:18.186+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:18.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:18.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:19.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:19.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:19.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:19.966+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:20.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:20.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:20.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:20.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:21.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:21.154+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:21.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:21.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:22.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:22.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:22.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:22.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:23.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:23.169+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:23.987 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:23.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:24.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:24.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:25.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:25.021+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:25.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:25.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:26.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:26.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:26.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:26.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:27.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:27.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:27.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:27.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:28.084 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:28.081+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:28.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:28.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:29.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:29.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:29.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:29.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:30.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:30.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:30.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:30.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:31.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:31.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:31.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:31.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:32.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:32.013+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:32.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:32.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:33.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:33.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:33.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:33.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:34.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:34.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:34.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:34.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:35.023 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:35.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:35.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:35.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:35.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:35.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:36.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:36.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:37.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:36.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:37.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:37.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:38.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:38.005+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:38.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:38.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:39.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:39.049+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:39.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:39.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:40.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:40.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:40.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:40.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:41.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:41.029+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:41.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:41.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:42.041 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:42.037+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:42.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:42.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:42.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:42.993+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:43.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:43.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:43.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:43.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:44.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:44.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:44.943 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:44.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:45.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:45.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:45.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:45.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:46.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:46.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:46.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:46.973+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:47.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:47.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:47.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:47.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:48.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:48.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:48.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:48.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:49.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:49.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:49.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:49.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:50.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:50.065+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:50.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:50.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:51.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:51.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:51.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:51.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:52.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:52.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:52.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:52.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:53.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:53.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:54.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:53.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:54.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:54.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:54.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:54.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:55.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:55.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:56.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:56.001+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:56.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:56.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:57.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:57.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:57.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:57.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:58.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:58.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:58.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:58.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:59.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:10:59.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:10:59.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:10:59.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:00.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:00.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:00.059 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:00.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:01.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:01.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:01.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:01.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:02.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:02.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:02.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:02.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:03.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:03.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:03.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:03.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:04.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:04.053+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:04.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:04.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:05.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:05.053+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:05.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:05.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:06.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:06.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:06.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:06.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:07.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:07.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:07.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:07.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:08.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:08.093+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:08.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:08.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:09.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:09.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:09.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:09.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:10.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:10.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:10.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:10.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:11.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:11.029+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:11.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:11.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:12.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:12.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:12.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:12.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:13.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:13.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:13.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:13.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:13.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:13.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:14.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:14.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:15.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:15.021+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:15.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:15.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:16.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:16.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:16.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:16.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:17.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:17.004+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:17.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:17.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:18.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:18.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:18.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:18.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:19.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:19.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:19.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:19.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:20.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:20.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:20.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:20.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:21.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:21.104+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:21.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:21.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:22.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:22.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:22.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:22.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:23.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:23.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:23.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:23.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:24.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:24.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:24.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:24.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:25.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:25.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:25.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:25.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:26.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:26.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:26.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:26.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:27.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:27.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:27.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:27.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:28.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:27.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:28.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:28.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:28.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:28.980+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:29.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:29.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:30.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:30.032+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:30.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:30.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:31.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:30.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:31.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:31.204+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:31.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:31.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:32.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:32.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:32.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:32.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:33.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:33.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:33.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:33.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:34.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:34.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:34.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:34.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:35.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:35.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:35.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:35.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:36.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:36.216+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:36.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:36.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:37.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:37.196+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:37.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:37.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:38.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:38.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:38.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:38.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:39.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:39.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:39.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:39.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:40.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:40.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:40.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:40.964+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:41.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:41.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:41.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:41.984+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:42.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:42.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:42.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:42.952+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:43.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:43.212+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:43.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:43.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:44.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:44.212+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:44.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:44.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:45.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:45.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:46.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:46.000+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:46.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:46.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:47.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:47.036+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:47.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:47.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:48.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:48.004+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:48.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:48.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:48.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:48.984+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:49.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:49.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:49.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:49.940+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:50.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:50.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:50.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:50.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:51.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:51.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:51.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:51.884+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:52.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:52.216+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:52.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:52.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:53.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:53.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:53.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:53.896+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:54.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:54.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:54.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:54.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:55.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:55.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:55.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:55.900+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:56.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:56.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:56.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:56.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:57.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:57.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:57.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:57.900+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:58.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:58.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:58.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:58.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:59.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:11:59.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:11:59.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:11:59.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:00.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:00.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:00.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:00.896+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:01.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:01.248+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:01.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:01.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:02.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:02.248+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:02.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:02.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:03.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:03.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:03.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:03.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:04.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:04.196+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:04.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:04.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:05.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:05.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:05.967 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:05.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:06.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:06.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:06.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:06.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:07.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:07.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:07.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:07.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:08.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:08.248+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:08.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:08.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:09.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:09.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:09.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:09.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:10.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:10.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:10.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:10.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:11.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:11.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:11.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:11.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:12.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:12.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:12.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:12.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:13.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:13.335+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:13.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:13.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:14.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:14.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:15.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:15.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:15.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:15.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:16.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:16.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:16.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:16.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:17.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:17.007+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:17.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:17.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:18.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:17.999+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:18.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:18.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:18.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:18.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:19.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:19.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:19.906 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:19.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:20.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:20.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:20.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:20.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:21.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:21.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:21.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:21.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:22.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:22.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:22.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:22.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:23.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:23.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:23.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:23.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:24.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:24.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:24.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:24.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:25.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:25.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:25.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:25.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:26.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:26.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:26.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:26.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:27.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:27.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:27.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:27.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:28.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:28.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:28.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:28.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:29.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:29.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:29.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:29.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:30.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:30.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:30.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:30.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:31.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:31.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:31.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:31.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:32.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:32.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:32.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:32.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:33.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:33.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:33.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:33.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:34.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:34.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:34.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:34.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:35.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:35.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:35.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:35.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:36.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:36.079+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:36.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:36.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:37.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:37.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:38.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:38.015+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:38.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:38.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:39.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:39.035+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:39.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:39.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:39.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:39.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:40.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:40.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:40.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:40.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:41.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:41.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:42.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:42.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:42.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:42.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:43.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:42.999+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:43.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:43.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:44.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:44.015+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:44.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:44.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:45.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:45.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:45.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:45.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:46.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:46.099+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:46.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:46.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:47.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:47.123+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:47.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:47.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:48.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:48.115+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:48.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:48.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:49.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:49.087+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:49.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:49.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:50.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:50.051+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:50.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:50.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:51.053 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:51.051+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:51.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:51.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:52.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:52.027+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:52.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:52.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:52.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:52.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:53.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:53.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:53.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:53.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:54.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:54.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:54.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:54.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:55.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:55.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:55.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:55.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:56.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:56.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:56.967 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:56.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:57.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:57.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:58.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:58.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:58.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:58.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:59.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:59.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:59.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:12:59.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:12:59.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:12:59.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:00.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:00.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:01.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:01.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:01.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:01.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:01.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:01.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:02.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:02.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:02.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:02.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:03.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:03.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:03.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:03.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:04.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:04.330+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:04.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:04.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:05.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:05.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:05.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:05.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:06.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:06.282+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:06.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:06.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:07.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:07.330+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:07.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:07.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:08.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:08.330+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:08.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:08.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:09.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:09.326+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:09.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:09.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:10.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:10.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:10.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:10.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:11.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:11.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:11.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:11.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:12.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:12.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:12.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:12.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:13.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:13.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:13.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:13.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:14.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:14.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:14.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:14.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:15.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:15.230+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:15.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:15.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:16.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:16.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:16.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:16.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:17.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:17.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:17.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:17.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:18.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:18.222+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:18.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:18.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:19.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:19.234+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:19.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:19.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:20.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:20.222+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:20.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:20.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:21.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:21.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:21.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:21.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:22.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:22.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:22.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:22.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:23.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:23.326+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:23.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:23.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:24.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:24.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:24.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:24.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:25.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:25.346+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:25.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:25.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:26.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:26.378+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:26.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:26.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:27.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:27.334+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:27.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:27.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:28.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:28.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:28.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:28.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:29.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:29.318+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:29.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:29.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:30.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:30.350+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:30.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:30.722+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:31.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:31.330+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:31.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:31.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:32.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:32.302+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:32.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:32.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:33.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:33.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:33.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:33.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:34.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:34.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:34.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:34.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:35.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:35.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:35.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:35.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:36.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:36.346+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:36.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:36.782+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:37.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:37.318+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:37.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:37.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:38.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:38.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:38.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:38.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:39.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:39.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:39.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:39.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:40.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:40.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:40.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:40.718+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:41.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:41.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:41.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:41.710+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:42.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:42.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:42.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:42.730+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:43.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:43.322+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:43.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:43.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:44.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:44.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:44.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:44.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:45.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:45.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:45.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:45.722+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:46.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:46.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:46.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:46.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:47.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:47.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:47.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:47.674+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:48.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:48.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:48.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:48.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:49.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:49.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:49.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:49.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:50.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:50.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:50.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:50.662+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:51.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:51.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:51.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:51.646+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:52.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:52.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:52.701 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:52.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:53.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:53.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:53.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:53.670+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:54.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:54.202+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:54.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:54.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:55.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:55.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:55.669 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:55.666+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:56.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:56.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:56.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:56.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:57.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:57.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:57.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:57.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:58.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:58.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:58.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:58.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:59.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:13:59.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:13:59.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:13:59.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:00.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:00.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:00.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:00.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:01.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:01.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:01.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:01.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:02.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:02.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:02.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:02.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:03.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:03.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:03.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:03.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:04.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:04.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:04.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:04.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:05.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:05.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:05.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:05.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:06.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:06.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:06.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:06.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:07.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:07.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:07.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:07.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:08.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:08.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:08.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:08.889+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:09.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:09.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:09.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:09.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:10.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:10.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:10.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:10.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:11.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:11.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:11.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:11.849+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:12.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:12.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:12.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:12.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:13.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:13.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:13.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:13.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:14.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:14.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:14.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:14.853+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:15.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:15.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:15.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:15.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:16.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:16.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:16.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:16.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:17.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:17.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:17.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:17.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:18.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:18.345+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:18.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:18.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:19.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:19.393+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:19.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:19.909+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:20.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:20.385+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:20.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:20.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:21.426 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:21.421+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:21.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:21.945+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:22.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:22.393+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:22.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:22.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:23.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:23.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:23.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:23.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:24.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:24.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:24.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:24.901+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:25.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:25.445+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:25.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:25.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:26.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:26.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:26.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:26.965+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:27.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:27.465+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:27.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:27.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:28.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:28.445+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:28.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:28.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:29.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:29.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:29.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:29.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:30.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:30.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:30.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:30.853+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:31.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:31.453+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:31.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:31.905+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:32.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:32.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:32.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:32.909+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:33.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:33.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:33.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:33.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:34.562 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:34.557+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:34.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:34.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:35.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:35.541+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:35.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:35.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:36.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:36.517+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:36.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:36.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:37.534 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:37.529+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:37.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:37.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:38.515 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:38.513+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:38.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:38.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:39.535 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:39.533+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:39.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:39.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:40.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:40.525+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:40.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:40.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:41.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:41.533+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:41.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:41.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:42.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:42.549+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:42.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:42.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:43.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:43.601+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:43.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:43.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:44.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:44.577+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:44.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:44.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:45.574 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:45.569+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:45.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:45.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:46.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:46.605+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:46.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:46.905+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:47.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:47.621+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:47.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:47.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:48.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:48.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:48.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:48.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:49.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:49.653+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:49.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:49.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:50.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:50.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:50.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:50.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:51.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:51.681+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:51.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:51.845+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:52.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:52.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:52.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:52.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:53.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:53.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:53.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:53.884+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:54.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:54.680+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:54.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:54.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:55.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:55.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:55.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:55.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:56.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:56.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:56.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:56.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:57.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:57.696+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:57.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:57.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:58.679 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:58.672+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:58.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:58.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:59.656 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:14:59.652+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:14:59.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:14:59.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:00.649 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:00.644+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:00.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:00.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:01.610 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:01.604+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:01.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:01.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:02.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:02.568+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:02.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:02.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:03.591 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:03.588+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:03.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:03.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:04.568 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:04.564+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:04.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:04.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:05.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:05.596+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:05.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:05.984+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:06.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:06.624+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:06.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:06.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:07.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:07.664+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:08.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:08.020+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:08.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:08.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:09.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:08.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:09.735 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:09.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:10.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:10.004+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:10.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:10.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:10.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:10.964+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:11.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:11.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:11.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:11.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:12.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:12.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:12.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:12.972+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:13.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:13.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:13.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:13.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:14.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:14.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:14.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:14.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:15.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:15.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:15.978 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:15.972+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:16.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:16.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:16.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:16.964+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:17.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:17.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:18.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:17.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:18.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:18.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:19.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:19.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:19.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:19.712+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:20.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:20.000+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:20.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:20.748+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:21.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:20.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:21.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:21.764+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:22.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:22.028+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:22.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:22.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:22.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:22.988+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:23.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:23.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:24.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:24.032+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:24.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:24.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:25.084 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:25.080+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:25.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:25.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:26.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:26.084+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:26.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:26.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:27.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:27.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:27.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:27.736+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:28.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:28.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:28.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:28.700+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:29.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:29.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:29.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:29.656+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:30.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:30.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:30.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:30.700+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:31.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:31.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:31.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:31.696+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:32.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:32.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:32.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:32.692+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:33.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:33.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:33.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:33.692+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:34.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:34.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:34.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:34.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:35.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:35.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:35.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:35.720+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:36.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:36.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:36.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:36.680+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:37.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:37.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:37.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:37.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:38.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:38.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:38.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:38.644+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:39.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:39.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:39.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:39.652+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:40.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:40.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:40.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:40.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:41.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:41.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:41.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:41.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:42.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:42.332+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:42.678 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:42.676+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:43.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:43.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:43.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:43.720+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:44.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:44.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:44.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:44.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:45.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:45.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:45.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:45.712+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:46.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:46.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:46.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:46.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:47.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:47.223+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:47.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:47.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:48.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:48.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:48.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:48.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:49.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:49.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:49.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:49.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:50.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:50.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:50.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:50.595+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:51.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:51.115+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:51.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:51.583+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:52.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:52.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:52.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:52.631+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:53.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:53.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:53.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:53.595+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:54.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:54.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:54.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:54.631+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:55.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:55.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:55.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:55.583+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:56.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:56.123+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:56.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:56.571+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:57.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:57.075+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:57.557 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:57.555+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:58.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:58.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:58.603 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:58.603+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:59.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:15:59.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:15:59.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:15:59.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:00.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:00.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:00.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:00.611+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:01.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:01.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:01.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:01.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:02.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:02.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:02.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:02.619+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:03.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:03.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:03.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:03.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:04.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:04.167+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:04.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:04.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:05.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:05.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:05.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:05.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:06.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:06.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:06.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:06.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:07.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:07.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:07.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:07.775+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:08.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:08.119+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:08.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:08.819+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:09.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:09.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:09.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:09.843+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:10.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:10.107+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:10.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:10.859+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:11.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:11.119+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:11.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:11.827+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:12.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:12.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:12.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:12.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:13.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:13.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:13.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:13.867+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:14.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:14.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:14.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:14.859+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:15.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:15.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:15.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:15.879+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:16.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:16.223+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:16.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:16.891+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:17.267 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:17.267+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:17.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:17.935+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:18.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:18.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:18.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:18.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:19.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:19.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:19.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:19.963+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:20.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:20.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:20.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:20.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:21.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:21.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:21.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:21.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:22.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:22.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:22.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:22.963+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:23.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:23.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:23.962 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:23.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:24.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:24.279+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:24.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:24.927+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:25.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:25.299+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:25.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:25.911+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:26.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:26.347+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:26.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:26.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:27.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:27.379+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:27.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:27.919+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:28.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:28.407+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:28.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:28.943+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:29.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:29.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:29.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:29.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:30.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:30.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:30.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:30.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:31.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:31.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:31.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:31.895+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:32.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:32.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:32.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:32.911+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:33.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:33.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:33.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:33.907+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:34.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:34.431+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:34.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:34.907+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:35.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:35.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:35.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:35.879+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:36.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:36.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:36.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:36.887+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:37.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:37.399+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:37.902 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:37.899+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:38.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:38.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:38.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:38.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:39.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:39.407+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:39.938 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:39.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:40.449 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:40.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:40.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:40.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:41.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:41.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:41.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:41.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:42.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:42.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:42.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:42.886+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:43.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:43.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:43.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:43.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:44.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:44.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:44.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:44.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:45.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:45.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:45.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:45.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:46.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:46.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:46.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:46.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:47.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:47.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:47.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:47.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:48.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:48.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:48.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:48.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:49.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:49.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:49.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:49.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:50.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:50.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:50.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:50.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:51.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:51.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:51.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:51.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:52.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:52.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:52.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:52.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:53.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:53.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:53.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:53.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:54.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:54.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:54.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:54.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:55.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:55.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:56.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:56.022+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:56.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:56.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:57.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:57.058+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:57.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:57.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:58.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:58.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:58.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:58.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:59.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:16:59.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:16:59.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:16:59.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:00.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:00.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:00.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:00.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:00.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:00.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:01.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:01.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:02.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:02.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:02.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:02.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:02.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:02.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:03.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:03.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:03.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:03.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:04.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:04.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:05.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:05.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:05.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:05.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:05.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:05.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:06.578 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:06.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:06.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:06.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:07.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:07.610+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:08.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:08.022+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:08.601 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:08.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:09.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:09.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:09.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:09.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:10.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:10.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:10.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:10.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:11.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:11.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:11.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:11.710+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:12.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:12.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:12.671 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:12.666+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:13.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:13.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:13.668 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:13.666+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:14.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:14.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:14.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:14.654+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:15.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:15.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:15.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:15.610+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:16.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:16.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:16.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:16.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:17.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:17.038+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:17.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:17.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:17.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:17.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:18.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:18.662+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:18.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:18.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:19.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:19.654+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:20.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:20.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:20.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:20.674+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:21.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:21.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:21.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:21.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:22.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:22.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:22.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:22.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:23.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:23.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:23.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:23.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:24.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:24.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:24.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:24.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:25.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:25.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:25.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:25.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:26.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:26.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:26.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:26.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:27.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:27.110+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:27.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:27.782+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:28.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:28.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:28.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:28.782+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:29.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:29.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:29.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:29.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:30.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:30.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:30.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:30.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:31.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:31.078+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:31.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:31.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:32.043 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:32.042+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:32.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:32.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:33.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:33.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:33.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:33.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:34.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:34.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:34.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:34.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:35.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:35.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:35.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:35.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:36.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:36.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:36.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:36.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:37.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:37.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:37.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:37.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:38.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:38.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:38.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:38.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:39.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:39.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:39.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:39.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:40.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:40.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:40.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:40.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:41.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:41.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:41.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:41.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:42.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:42.065+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:42.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:42.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:43.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:43.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:43.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:43.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:44.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:44.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:44.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:44.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:45.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:45.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:45.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:45.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:46.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:46.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:46.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:46.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:47.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:47.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:47.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:47.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:48.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:48.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:48.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:48.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:49.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:49.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:49.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:49.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:50.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:50.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:50.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:50.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:51.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:51.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:51.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:51.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:52.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:52.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:52.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:52.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:53.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:53.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:53.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:53.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:54.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:54.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:54.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:54.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:55.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:55.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:55.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:55.749+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:56.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:56.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:56.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:56.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:57.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:57.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:57.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:57.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:58.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:58.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:58.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:58.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:59.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:17:59.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:17:59.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:17:59.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:00.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:00.021+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:00.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:00.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:01.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:01.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:01.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:01.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:02.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:02.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:02.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:02.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:03.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:03.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:03.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:03.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:04.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:04.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:04.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:04.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:05.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:05.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:05.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:05.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:06.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:06.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:06.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:06.749+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:07.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:07.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:07.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:07.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:08.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:08.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:08.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:08.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:09.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:09.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:09.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:09.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:10.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:10.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:10.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:10.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:11.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:11.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:11.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:11.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:12.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:12.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:12.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:12.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:13.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:13.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:13.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:13.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:14.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:14.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:14.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:14.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:15.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:15.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:15.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:15.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:16.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:16.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:16.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:16.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:17.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:17.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:17.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:17.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:18.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:18.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:18.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:18.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:19.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:19.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:19.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:19.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:20.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:20.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:20.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:20.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:21.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:21.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:21.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:21.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:22.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:22.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:22.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:22.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:23.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:23.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:23.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:23.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:24.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:24.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:24.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:24.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:25.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:25.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:25.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:25.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:26.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:26.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:26.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:26.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:27.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:27.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:28.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:28.016+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:28.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:28.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:29.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:29.036+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:29.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:29.224+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:30.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:30.032+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:30.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:30.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:31.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:31.048+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:31.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:31.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:32.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:32.032+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:32.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:32.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:33.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:33.048+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:33.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:33.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:34.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:34.004+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:34.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:34.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:34.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:34.988+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:35.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:35.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:35.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:35.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:36.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:36.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:36.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:36.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:37.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:37.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:37.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:37.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:38.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:38.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:38.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:38.988+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:39.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:39.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:39.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:39.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:40.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:40.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:40.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:40.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:41.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:41.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:42.017 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:42.012+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:42.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:42.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:42.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:42.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:43.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:43.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:43.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:43.988+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:44.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:44.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:44.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:44.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:45.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:45.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:46.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:46.020+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:46.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:46.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:47.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:47.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:47.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:47.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:47.982 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:47.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:48.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:48.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:49.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:49.000+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:49.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:49.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:50.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:50.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:50.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:50.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:51.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:51.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:51.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:51.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:52.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:52.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:52.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:52.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:53.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:53.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:53.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:53.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:54.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:54.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:54.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:54.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:55.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:55.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:55.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:55.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:56.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:56.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:56.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:56.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:57.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:57.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:57.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:57.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:58.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:58.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:58.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:58.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:59.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:18:59.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:18:59.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:18:59.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:00.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:00.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:00.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:00.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:01.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:01.056+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:01.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:01.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:02.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:02.040+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:02.545 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:02.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:03.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:03.064+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:03.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:03.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:04.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:04.016+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:04.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:04.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:05.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:05.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:05.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:05.560+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:06.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:06.048+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:06.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:06.556+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:07.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:07.068+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:07.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:07.560+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:08.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:08.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:08.596 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:08.596+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:09.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:09.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:09.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:09.568+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:10.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:10.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:10.580 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:10.580+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:11.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:11.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:11.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:11.556+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:12.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:12.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:12.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:12.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:13.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:13.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:13.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:13.520+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:14.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:14.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:14.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:14.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:15.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:15.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:15.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:15.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:16.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:16.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:16.506 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:16.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:17.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:17.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:17.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:17.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:18.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:18.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:18.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:18.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:19.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:19.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:19.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:19.475+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:20.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:20.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:20.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:20.427+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:21.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:21.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:21.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:21.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:22.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:22.055+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:22.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:22.495+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:23.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:23.107+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:23.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:23.479+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:24.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:24.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:24.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:24.471+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:25.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:25.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:25.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:25.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:26.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:26.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:26.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:26.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:27.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:27.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:27.516 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:27.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:28.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:28.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:28.515 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:28.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:29.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:29.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:29.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:29.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:30.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:30.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:30.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:30.459+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:31.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:31.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:31.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:31.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:32.172 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:32.171+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:32.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:32.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:33.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:33.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:33.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:33.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:34.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:34.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:34.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:34.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:35.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:35.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:35.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:35.379+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:36.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:36.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:36.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:36.335+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:37.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:37.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:37.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:37.347+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:38.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:38.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:38.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:38.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:39.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:39.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:39.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:39.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:40.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:40.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:40.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:40.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:41.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:41.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:41.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:41.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:42.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:42.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:42.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:42.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:43.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:43.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:43.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:43.267+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:44.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:44.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:44.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:44.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:45.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:45.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:45.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:45.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:46.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:46.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:46.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:46.251+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:47.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:47.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:47.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:47.279+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:48.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:48.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:48.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:48.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:49.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:49.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:49.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:49.271+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:50.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:50.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:50.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:50.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:51.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:51.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:51.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:51.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:52.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:52.155+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:52.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:52.291+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:53.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:53.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:53.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:53.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:54.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:54.155+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:54.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:54.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:55.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:55.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:55.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:55.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:56.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:56.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:56.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:56.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:57.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:57.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:57.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:57.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:58.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:58.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:58.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:58.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:59.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:19:59.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:19:59.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:19:59.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:00.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:00.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:00.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:00.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:01.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:01.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:01.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:01.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:02.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:02.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:02.267 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:02.267+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:03.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:03.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:03.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:03.299+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:04.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:04.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:04.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:04.343+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:05.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:05.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:05.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:05.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:06.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:06.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:06.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:06.271+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:07.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:07.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:07.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:07.323+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:08.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:08.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:08.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:08.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:09.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:09.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:09.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:09.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:10.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:10.155+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:10.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:10.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:10.750 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T23:20:10.751+0000 7fd5e8fc6700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fd5cc058290 msgr2=0x7fd5cc0d10d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T23:20:10.756 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-11T23:20:10.755+0000 7fd5e8fc6700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x560328ce14f0 msgr2=0x7fd5cc032ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-11T23:20:11.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:11.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:11.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:11.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:12.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:12.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:12.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:12.434+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:13.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:13.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:13.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:13.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:14.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:14.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:14.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:14.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:15.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:15.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:15.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:15.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:16.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:16.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:16.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:16.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:17.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:17.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:17.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:17.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:18.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:18.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:18.369 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:18.366+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:19.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:19.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:19.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:19.350+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:20.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:20.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:20.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:20.350+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:21.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:21.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:21.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:21.394+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:22.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:22.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:22.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:22.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:23.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:23.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:23.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:23.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:24.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:24.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:24.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:24.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:25.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:25.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:25.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:25.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:26.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:26.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:26.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:26.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:27.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:27.234+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:27.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:27.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:28.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:28.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:28.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:28.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:29.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:29.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:29.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:29.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:30.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:30.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:30.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:30.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:31.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:31.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:31.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:31.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:32.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:32.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:32.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:32.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:33.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:33.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:33.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:33.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:34.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:34.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:34.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:34.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:35.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:35.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:35.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:35.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:36.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:36.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:36.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:36.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:37.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:37.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:37.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:37.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:38.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:38.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:38.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:38.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:39.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:39.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:39.466 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:39.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:40.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:40.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:40.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:40.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:41.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:41.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:41.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:41.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:42.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:42.154+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:42.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:42.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:43.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:43.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:43.563 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:43.562+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:44.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:44.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:44.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:44.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:45.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:45.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:45.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:45.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:46.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:46.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:46.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:46.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:47.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:47.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:47.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:47.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:48.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:48.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:48.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:48.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:49.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:49.010+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:49.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:49.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:49.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:49.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:50.580 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:50.578+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:50.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:50.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:51.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:51.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:51.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:51.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:52.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:52.590+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:52.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:52.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:53.595 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:53.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:53.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:53.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:54.563 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:54.562+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:54.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:54.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:55.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:55.558+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:55.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:55.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:56.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:56.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:56.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:56.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:57.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:57.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:57.938 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:57.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:58.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:58.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:58.968 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:20:58.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:20:59.571 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:20:59.570+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:00.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:00.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:00.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:00.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:01.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:01.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:01.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:01.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:02.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:02.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:02.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:02.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:03.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:03.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:03.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:03.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:04.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:04.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:04.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:04.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:05.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:05.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:05.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:05.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:06.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:06.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:06.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:06.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:07.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:07.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:07.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:07.549+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:08.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:08.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:08.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:08.581+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:09.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:09.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:09.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:09.625+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:10.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:10.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:10.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:10.629+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:11.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:11.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:11.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:11.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:12.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:12.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:12.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:12.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:13.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:13.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:13.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:13.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:14.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:14.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:14.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:14.677+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:15.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:15.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:15.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:15.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:16.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:16.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:16.694 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:16.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:17.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:17.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:17.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:17.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:18.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:18.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:18.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:18.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:19.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:19.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:19.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:19.673+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:20.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:20.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:20.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:20.677+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:21.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:21.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:21.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:21.657+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:22.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:22.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:22.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:22.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:23.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:23.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:23.619 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:23.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:24.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:24.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:24.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:24.569+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:25.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:25.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:25.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:25.533+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:26.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:26.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:26.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:26.529+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:27.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:27.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:27.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:27.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:28.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:28.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:28.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:28.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:29.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:29.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:29.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:29.605+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:30.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:30.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:30.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:30.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:31.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:31.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:31.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:31.637+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:32.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:32.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:32.667 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:32.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:33.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:33.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:33.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:33.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:34.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:34.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:34.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:34.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:35.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:35.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:35.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:35.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:36.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:36.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:36.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:36.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:37.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:37.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:37.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:37.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:38.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:38.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:38.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:38.673+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:39.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:39.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:39.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:39.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:40.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:40.185+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:40.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:40.629+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:41.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:41.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:41.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:41.673+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:42.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:42.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:42.671 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:42.669+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:43.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:43.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:43.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:43.673+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:44.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:44.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:44.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:44.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:45.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:45.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:45.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:45.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:46.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:46.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:46.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:46.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:47.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:47.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:47.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:47.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:48.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:48.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:48.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:48.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:49.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:49.345+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:49.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:49.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:50.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:50.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:50.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:50.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:51.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:51.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:51.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:51.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:52.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:52.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:52.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:52.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:53.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:53.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:53.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:53.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:54.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:54.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:54.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:54.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:55.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:55.433+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:55.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:55.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:56.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:56.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:56.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:56.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:57.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:57.525+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:57.684 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:57.681+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:58.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:58.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:58.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:58.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:59.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:21:59.532+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:21:59.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:21:59.608+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:00.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:00.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:00.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:00.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:01.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:01.556+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:01.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:01.584+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:02.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:02.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:02.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:02.576+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:03.584 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:03.580+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:03.603 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:03.600+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:04.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:04.552+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:04.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:04.620+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:05.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:05.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:05.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:05.608+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:06.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:06.528+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:06.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:06.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:07.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:07.556+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:07.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:07.664+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:08.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:08.580+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:08.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:08.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:09.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:09.556+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:09.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:09.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:10.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:10.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:10.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:10.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:11.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:11.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:11.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:11.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:12.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:12.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:12.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:12.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:13.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:13.580+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:13.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:13.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:14.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:14.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:14.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:14.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:15.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:15.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:15.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:15.720+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:16.595 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:16.592+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:16.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:16.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:17.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:17.612+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:17.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:17.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:18.668 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:18.664+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:18.684 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:18.680+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:19.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:19.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:19.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:19.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:20.667 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:20.664+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:20.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:20.668+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:21.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:21.616+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:21.678 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:21.672+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:22.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:22.608+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:22.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:22.700+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:23.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:23.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:23.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:23.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:24.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:24.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:24.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:24.712+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:25.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:25.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:25.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:25.736+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:26.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:26.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:26.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:26.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:27.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:27.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:27.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:27.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:28.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:28.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:28.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:28.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:29.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:29.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:29.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:29.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:30.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:30.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:30.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:30.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:31.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:31.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:31.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:31.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:32.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:32.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:32.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:32.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:33.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:33.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:33.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:33.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:34.679 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:34.676+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:34.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:34.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:35.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:35.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:35.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:35.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:36.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:36.652+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:36.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:36.656+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:37.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:37.620+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:37.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:37.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:38.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:38.628+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:38.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:38.644+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:39.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:39.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:39.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:39.636+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:40.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:40.628+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:40.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:40.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:41.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:41.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:41.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:41.676+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:42.626 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:42.624+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:42.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:42.628+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:43.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:43.628+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:43.660 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:43.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:44.627 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:44.628+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:44.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:44.668+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:45.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:45.644+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:45.668 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:45.668+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:46.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:46.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:46.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:46.692+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:47.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:47.652+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:47.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:47.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:48.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:48.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:48.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:48.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:49.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:49.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:49.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:49.732+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:50.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:50.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:50.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:50.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:51.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:51.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:51.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:51.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:52.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:52.719+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:52.752 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:52.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:53.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:53.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:53.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:53.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:54.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:54.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:54.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:54.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:55.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:55.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:55.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:55.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:56.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:56.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:56.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:56.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:57.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:57.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:57.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:57.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:58.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:58.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:58.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:58.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:59.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:22:59.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:22:59.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:22:59.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:00.750 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:00.747+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:00.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:00.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:01.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:01.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:01.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:01.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:02.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:02.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:02.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:02.783+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:03.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:03.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:03.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:03.759+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:04.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:04.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:04.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:04.783+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:05.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:05.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:05.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:05.775+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:06.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:06.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:06.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:06.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:07.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:07.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:07.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:07.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:08.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:08.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:08.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:08.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:09.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:09.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:09.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:09.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:10.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:10.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:10.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:10.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:11.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:11.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:11.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:11.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:12.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:12.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:12.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:12.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:13.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:13.719+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:13.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:13.799+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:14.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:14.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:14.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:14.831+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:15.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:15.651+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:15.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:15.863+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:16.636 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:16.635+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:16.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:16.867+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:17.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:17.683+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:17.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:17.891+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:18.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:18.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:18.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:18.915+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:19.653 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:19.651+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:19.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:19.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:20.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:20.663+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:20.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:20.911+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:21.625 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:21.623+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:21.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:21.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:22.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:22.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:22.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:22.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:23.580 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:23.579+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:23.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:23.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:24.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:24.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:24.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:24.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:25.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:25.555+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:25.933 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:25.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:26.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:26.547+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:26.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:26.903+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:27.598 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:27.595+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:27.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:27.915+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:28.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:28.639+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:28.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:28.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:29.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:29.643+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:29.926 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:29.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:30.603 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:30.603+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:30.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:30.943+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:31.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:31.603+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:31.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:31.899+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:32.653 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:32.651+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:32.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:32.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:33.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:33.651+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:33.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:33.919+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:34.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:34.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:34.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:34.943+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:35.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:35.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:35.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:35.899+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:36.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:36.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:36.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:36.943+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:37.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:37.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:37.908 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:37.907+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:38.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:38.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:38.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:38.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:39.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:39.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:39.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:39.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:40.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:40.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:40.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:40.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:41.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:41.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:41.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:41.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:42.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:42.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:42.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:42.907+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:43.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:43.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:43.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:43.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:44.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:44.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:44.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:44.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:45.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:45.718+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:45.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:45.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:46.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:46.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:46.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:46.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:47.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:47.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:47.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:47.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:48.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:48.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:48.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:48.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:49.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:49.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:49.917 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:49.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:50.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:50.754+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:50.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:50.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:51.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:51.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:51.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:51.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:52.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:52.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:52.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:52.930+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:53.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:53.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:53.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:53.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:54.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:54.662+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:54.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:54.886+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:55.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:55.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:55.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:55.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:56.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:56.662+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:56.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:56.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:57.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:57.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:57.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:57.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:58.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:58.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:58.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:58.774+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:59.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:23:59.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:23:59.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:23:59.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:00.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:00.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:00.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:00.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:01.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:01.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:01.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:01.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:02.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:02.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:02.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:02.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:03.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:03.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:03.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:03.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:04.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:04.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:04.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:04.718+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:05.641 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:05.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:05.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:05.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:06.599 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:06.598+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:06.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:06.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:07.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:07.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:07.649 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:07.646+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:08.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:08.562+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:08.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:08.658+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:09.610 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:09.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:09.694 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:09.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:10.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:10.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:10.691 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:10.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:11.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:11.546+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:11.684 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:11.682+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:12.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:12.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:12.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:12.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:13.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:13.538+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:13.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:13.646+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:14.540 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:14.538+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:14.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:14.618+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:15.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:15.494+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:15.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:15.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:16.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:16.494+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:16.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:16.590+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:17.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:17.506+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:17.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:17.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:18.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:18.474+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:18.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:18.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:19.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:19.502+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:19.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:19.614+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:20.538 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:20.534+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:20.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:20.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:21.545 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:21.542+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:21.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:21.658+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:22.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:22.518+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:22.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:22.678+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:23.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:23.566+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:23.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:23.654+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:24.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:24.542+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:24.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:24.686+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:25.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:25.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:25.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:25.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:26.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:26.546+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:26.694 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:26.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:27.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:27.522+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:27.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:27.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:28.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:28.518+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:28.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:28.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:29.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:29.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:29.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:29.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:30.543 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:30.542+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:30.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:30.706+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:31.535 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:31.534+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:31.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:31.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:32.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:32.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:32.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:32.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:33.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:33.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:33.671 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:33.670+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:34.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:34.506+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:34.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:34.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:35.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:35.482+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:35.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:35.650+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:36.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:36.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:36.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:36.609+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:37.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:37.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:37.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:37.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:38.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:38.469+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:38.593 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:38.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:39.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:39.497+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:39.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:39.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:40.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:40.529+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:40.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:40.573+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:41.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:41.541+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:41.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:41.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:42.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:42.529+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:42.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:42.545+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:43.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:43.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:43.543 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:43.541+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:44.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:44.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:44.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:44.549+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:45.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:45.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:45.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:45.545+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:46.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:46.521+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:46.588 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:46.585+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:47.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:47.557+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:47.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:47.625+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:48.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:48.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:48.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:48.593+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:49.557 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:49.553+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:49.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:49.557+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:50.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:50.565+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:50.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:50.593+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:51.580 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:51.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:51.613 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:51.609+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:52.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:52.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:52.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:52.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:53.580 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:53.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:53.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:53.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:54.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:54.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:54.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:54.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:55.571 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:55.569+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:55.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:55.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:56.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:56.581+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:56.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:56.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:57.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:57.533+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:57.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:57.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:58.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:58.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:58.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:58.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:59.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:24:59.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:24:59.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:24:59.749+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:00.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:00.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:00.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:00.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:01.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:01.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:01.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:01.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:02.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:02.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:02.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:02.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:03.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:03.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:03.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:03.629+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:04.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:04.433+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:04.603 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:04.601+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:05.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:05.417+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:05.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:05.649+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:06.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:06.373+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:06.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:06.625+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:07.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:07.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:07.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:07.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:08.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:08.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:08.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:08.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:09.402 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:09.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:09.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:09.617+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:10.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:10.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:10.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:10.641+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:11.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:11.433+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:11.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:11.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:12.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:12.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:12.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:12.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:13.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:13.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:13.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:13.665+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:14.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:14.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:14.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:14.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:15.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:15.461+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:15.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:15.665+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:16.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:16.497+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:16.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:16.689+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:17.458 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:17.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:17.649 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:17.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:18.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:18.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:18.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:18.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:19.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:19.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:19.635 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:19.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:20.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:20.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:20.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:20.653+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:21.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:21.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:21.613 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:21.609+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:22.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:22.549+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:22.591 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:22.589+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:23.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:23.549+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:23.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:23.553+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:24.534 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:24.533+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:24.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:24.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:25.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:25.501+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:25.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:25.541+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:26.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:26.473+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:26.541 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:26.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:27.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:27.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:27.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:27.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:28.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:28.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:28.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:28.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:29.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:29.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:29.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:29.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:30.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:30.476+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:30.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:30.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:31.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:31.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:31.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:31.520+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:32.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:32.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:32.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:32.484+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:33.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:33.424+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:33.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:33.500+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:34.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:34.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:34.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:34.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:35.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:35.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:35.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:35.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:36.423 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:36.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:36.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:36.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:37.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:37.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:37.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:37.592+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:38.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:38.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:38.633 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:38.628+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:39.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:39.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:39.618 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:39.612+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:40.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:40.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:40.660 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:40.656+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:41.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:41.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:41.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:41.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:42.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:42.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:42.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:42.632+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:43.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:43.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:43.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:43.628+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:44.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:44.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:44.636 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:44.632+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:45.512 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:45.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:45.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:45.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:46.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:46.532+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:46.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:46.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:47.564 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:47.560+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:47.618 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:47.612+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:48.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:48.588+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:48.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:48.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:49.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:49.572+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:49.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:49.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:50.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:50.560+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:50.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:50.584+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:51.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:51.552+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:51.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:51.564+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:52.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:52.552+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:52.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:52.588+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:53.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:53.508+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:53.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:53.584+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:54.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:54.492+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:54.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:54.576+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:55.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:55.456+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:55.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:55.572+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:56.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:56.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:56.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:56.552+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:57.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:57.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:57.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:57.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:58.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:58.472+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:58.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:58.592+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:59.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:25:59.512+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:25:59.596 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:25:59.592+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:00.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:00.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:00.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:00.552+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:01.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:01.484+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:01.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:01.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:02.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:02.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:02.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:02.536+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:03.470 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:03.464+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:03.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:03.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:04.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:04.424+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:04.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:04.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:05.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:05.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:05.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:05.536+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:06.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:06.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:06.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:06.496+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:07.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:07.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:07.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:07.492+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:08.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:08.452+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:08.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:08.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:09.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:09.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:09.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:09.496+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:10.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:10.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:10.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:10.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:11.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:11.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:11.538 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:11.540+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:12.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:12.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:12.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:12.560+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:13.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:13.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:13.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:13.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:14.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:14.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:14.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:14.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:15.366 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:15.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:15.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:15.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:16.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:16.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:16.643 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:16.644+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:17.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:17.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:17.631 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:17.632+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:18.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:18.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:18.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:18.652+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:19.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:19.528+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:19.635 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:19.636+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:20.516 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:20.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:20.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:20.664+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:21.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:21.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:21.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:21.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:22.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:22.559+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:22.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:22.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:23.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:23.567+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:23.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:23.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:24.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:24.571+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:24.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:24.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:25.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:25.551+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:25.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:25.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:26.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:26.543+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:26.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:26.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:27.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:27.523+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:27.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:27.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:28.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:28.471+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:28.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:28.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:29.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:29.471+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:29.687 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:29.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:30.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:30.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:30.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:30.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:31.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:31.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:31.687 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:31.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:32.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:32.415+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:32.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:32.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:33.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:33.415+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:33.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:33.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:34.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:34.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:34.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:34.651+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:35.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:35.379+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:35.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:35.659+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:36.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:36.367+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:36.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:36.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:37.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:37.411+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:37.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:37.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:38.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:38.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:38.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:38.691+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:39.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:39.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:39.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:39.663+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:40.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:40.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:40.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:40.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:41.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:41.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:41.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:41.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:42.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:42.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:42.649 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:42.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:43.456 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:43.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:43.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:43.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:44.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:44.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:44.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:44.659+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:45.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:45.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:45.683 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:45.683+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:46.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:46.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:46.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:46.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:47.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:47.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:47.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:47.723+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:48.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:48.507+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:48.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:48.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:49.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:49.507+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:49.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:49.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:50.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:50.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:50.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:50.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:51.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:51.519+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:51.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:51.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:52.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:52.471+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:52.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:52.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:53.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:53.471+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:53.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:53.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:54.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:54.463+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:54.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:54.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:55.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:55.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:55.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:55.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:56.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:56.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:56.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:56.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:57.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:57.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:57.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:57.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:58.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:58.423+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:58.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:58.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:59.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:26:59.431+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:26:59.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:26:59.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:00.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:00.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:00.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:00.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:01.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:01.427+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:01.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:01.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:02.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:02.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:02.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:02.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:03.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:03.359+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:03.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:03.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:04.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:04.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:04.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:04.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:05.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:05.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:05.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:05.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:06.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:06.371+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:06.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:06.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:07.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:07.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:07.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:07.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:08.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:08.367+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:08.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:08.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:09.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:09.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:09.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:09.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:10.358 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:10.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:10.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:10.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:11.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:11.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:11.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:11.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:12.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:12.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:12.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:12.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:13.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:13.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:13.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:13.954+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:14.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:14.354+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:14.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:14.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:15.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:15.322+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:15.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:15.934+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:16.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:16.274+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:16.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:16.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:17.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:17.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:17.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:17.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:18.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:18.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:18.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:18.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:19.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:19.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:19.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:19.918+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:20.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:20.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:20.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:20.966+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:21.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:21.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:21.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:21.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:22.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:22.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:22.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:22.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:23.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:23.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:23.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:23.970+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:24.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:24.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:24.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:24.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:25.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:25.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:26.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:26.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:26.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:26.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:27.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:27.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:27.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:27.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:28.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:28.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:28.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:28.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:29.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:29.050+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:29.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:29.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:30.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:30.062+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:30.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:30.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:31.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:31.086+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:31.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:31.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:32.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:32.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:32.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:32.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:33.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:33.054+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:33.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:33.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:34.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:34.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:34.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:34.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:35.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:35.102+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:35.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:35.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:36.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:36.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:36.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:36.078+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:37.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:37.086+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:37.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:37.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:38.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:38.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:38.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:38.130+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:39.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:39.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:39.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:39.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:40.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:40.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:40.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:40.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:41.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:41.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:41.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:41.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:42.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:42.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:42.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:42.198+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:43.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:43.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:43.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:43.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:44.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:44.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:44.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:44.174+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:45.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:45.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:45.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:45.154+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:46.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:46.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:46.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:46.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:47.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:47.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:47.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:47.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:48.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:48.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:48.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:48.182+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:49.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:49.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:49.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:49.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:50.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:50.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:50.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:50.190+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:51.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:51.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:51.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:51.174+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:52.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:52.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:52.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:52.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:53.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:53.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:53.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:53.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:54.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:54.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:54.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:54.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:55.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:55.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:55.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:55.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:56.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:56.078+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:56.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:56.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:57.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:57.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:57.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:57.186+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:58.083 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:58.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:58.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:58.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:59.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:27:59.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:27:59.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:27:59.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:00.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:00.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:00.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:00.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:01.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:01.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:01.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:01.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:02.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:02.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:02.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:02.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:03.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:03.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:03.128 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:03.126+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:04.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:04.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:04.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:04.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:05.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:05.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:05.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:05.110+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:06.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:06.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:06.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:06.125+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:07.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:07.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:07.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:07.125+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:08.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:08.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:08.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:08.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:09.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:09.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:09.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:09.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:10.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:10.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:10.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:10.157+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:11.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:11.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:11.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:11.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:12.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:12.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:12.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:12.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:13.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:13.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:13.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:13.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:14.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:14.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:14.134 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:14.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:15.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:15.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:15.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:15.125+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:16.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:16.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:16.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:16.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:17.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:17.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:17.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:17.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:18.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:18.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:18.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:18.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:19.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:19.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:19.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:19.229+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:20.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:20.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:20.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:20.265+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:21.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:21.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:21.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:21.229+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:22.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:22.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:22.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:22.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:23.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:23.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:23.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:23.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:24.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:24.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:24.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:24.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:25.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:25.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:25.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:25.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:26.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:26.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:26.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:26.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:27.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:27.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:27.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:27.157+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:28.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:28.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:28.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:28.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:29.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:29.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:29.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:29.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:30.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:30.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:30.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:30.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:31.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:31.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:31.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:31.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:32.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:32.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:32.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:32.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:33.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:33.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:33.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:33.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:34.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:34.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:34.176 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:34.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:35.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:35.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:35.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:35.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:36.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:36.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:36.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:36.141+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:37.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:37.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:37.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:37.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:38.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:38.105+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:38.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:38.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:39.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:39.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:39.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:39.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:40.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:40.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:40.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:40.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:41.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:41.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:41.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:41.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:42.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:42.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:42.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:42.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:43.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:43.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:43.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:43.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:44.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:44.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:44.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:44.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:45.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:45.057+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:45.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:45.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:46.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:46.069+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:46.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:46.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:47.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:47.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:47.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:47.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:48.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:48.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:48.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:48.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:49.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:49.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:49.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:49.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:50.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:50.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:50.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:50.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:51.084 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:51.081+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:51.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:51.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:52.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:52.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:52.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:52.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:53.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:53.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:53.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:53.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:54.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:54.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:54.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:54.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:55.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:55.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:55.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:55.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:56.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:56.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:56.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:56.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:57.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:57.105+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:57.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:57.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:58.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:58.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:58.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:58.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:59.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:28:59.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:28:59.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:28:59.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:00.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:00.084+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:00.335 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:00.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:01.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:01.040+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:01.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:01.376+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:02.060 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:02.056+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:02.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:02.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:03.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:03.052+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:03.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:03.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:04.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:04.080+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:04.337 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:04.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:05.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:05.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:05.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:05.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:06.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:06.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:06.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:06.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:07.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:07.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:07.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:07.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:08.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:08.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:08.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:08.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:09.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:09.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:09.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:09.360+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:10.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:10.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:10.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:10.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:11.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:11.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:11.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:11.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:12.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:12.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:12.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:12.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:13.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:13.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:13.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:13.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:14.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:14.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:14.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:14.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:15.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:15.360+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:15.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:15.528+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:16.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:16.348+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:16.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:16.576+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:17.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:17.368+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:17.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:17.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:18.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:18.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:18.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:18.588+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:19.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:19.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:19.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:19.548+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:20.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:20.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:20.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:20.568+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:21.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:21.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:21.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:21.572+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:22.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:22.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:22.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:22.564+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:23.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:23.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:23.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:23.604+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:24.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:24.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:24.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:24.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:25.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:25.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:25.631 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:25.628+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:26.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:26.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:26.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:26.664+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:27.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:27.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:27.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:27.640+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:28.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:28.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:28.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:28.668+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:29.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:29.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:29.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:29.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:30.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:30.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:30.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:30.640+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:31.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:31.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:31.599 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:31.596+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:32.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:32.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:32.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:32.584+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:33.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:33.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:33.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:33.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:34.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:34.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:34.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:34.600+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:35.126 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:35.124+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:35.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:35.636+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:36.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:36.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:36.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:36.624+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:37.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:37.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:37.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:37.640+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:38.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:38.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:38.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:38.612+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:39.128 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:39.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:39.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:39.584+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:40.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:40.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:40.612 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:40.612+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:41.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:41.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:41.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:41.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:42.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:42.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:42.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:42.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:43.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:43.032+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:43.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:43.672+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:44.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:44.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:44.640 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:44.640+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:45.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:45.036+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:45.632 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:45.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:46.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:46.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:46.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:46.616+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:47.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:47.064+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:47.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:47.664+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:48.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:48.104+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:48.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:48.656+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:49.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:49.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:49.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:49.636+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:50.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:50.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:50.618 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:50.616+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:51.128 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:51.127+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:51.619 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:51.619+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:52.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:52.123+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:52.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:52.603+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:53.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:53.123+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:53.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:53.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:54.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:54.115+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:54.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:54.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:55.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:55.091+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:55.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:55.619+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:56.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:56.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:56.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:56.603+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:57.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:57.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:57.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:57.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:58.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:58.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:58.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:58.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:59.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:29:59.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:29:59.679 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:29:59.679+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:00.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:00.075+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:00.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:00.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:01.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:01.095+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:01.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:01.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:02.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:02.083+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:02.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:02.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:03.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:03.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:03.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:03.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:04.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:04.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:04.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:04.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:05.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:05.031+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:05.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:05.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:06.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:06.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:06.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:06.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:06.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:06.979+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:07.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:07.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:07.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:07.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:08.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:08.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:08.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:08.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:09.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:09.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:09.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:09.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:10.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:10.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:10.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:10.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:11.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:11.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:11.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:11.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:12.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:12.783+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:12.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:12.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:13.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:13.775+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:13.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:13.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:14.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:14.783+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:14.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:14.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:15.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:15.747+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:15.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:15.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:16.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:16.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:16.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:16.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:17.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:17.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:18.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:17.999+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:18.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:18.827+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:19.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:19.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:19.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:19.867+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:20.093 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:20.091+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:20.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:20.903+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:21.093 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:21.091+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:21.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:21.895+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:22.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:22.051+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:22.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:22.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:23.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:23.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:23.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:23.943+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:24.060 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:24.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:24.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:24.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:25.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:25.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:25.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:25.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:26.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:26.031+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:27.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:26.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:27.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:27.051+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:27.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:27.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:28.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:28.047+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:28.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:28.967+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:29.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:29.035+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:29.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:29.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:30.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:30.087+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:31.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:31.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:31.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:31.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:31.964 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:31.963+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:32.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:32.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:32.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:32.979+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:32.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:32.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:33.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:33.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:34.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:34.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:34.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:34.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:35.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:35.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:35.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:35.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:36.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:36.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:36.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:36.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:37.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:37.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:37.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:37.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:38.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:38.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:39.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:38.999+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:39.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:39.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:39.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:39.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:40.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:40.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:41.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:40.999+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:41.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:41.035+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:41.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:41.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:42.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:42.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:42.966 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:42.963+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:42.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:42.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:43.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:43.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:43.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:43.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:44.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:44.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:44.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:44.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:45.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:45.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:45.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:45.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:46.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:46.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:46.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:46.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:47.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:47.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:47.917 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:47.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:48.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:48.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:48.917 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:48.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:49.874 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:49.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:49.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:49.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:50.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:50.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:50.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:50.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:51.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:51.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:51.874 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:51.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:52.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:52.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:52.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:52.914+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:53.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:53.886+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:53.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:53.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:54.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:54.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:54.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:54.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:55.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:55.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:55.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:55.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:56.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:56.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:56.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:56.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:57.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:57.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:57.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:57.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:58.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:58.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:58.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:58.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:59.866 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:30:59.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:30:59.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:30:59.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:00.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:00.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:00.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:00.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:01.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:01.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:01.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:01.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:02.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:02.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:02.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:02.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:03.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:03.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:03.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:03.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:04.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:04.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:04.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:04.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:05.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:05.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:05.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:05.942+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:06.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:06.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:06.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:06.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:07.808 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:07.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:07.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:07.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:08.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:08.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:08.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:08.950+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:09.745 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:09.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:09.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:09.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:10.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:10.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:10.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:10.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:11.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:11.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:11.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:11.982+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:12.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:12.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:13.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:12.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:13.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:13.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:14.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:14.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:14.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:14.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:15.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:15.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:15.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:15.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:16.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:16.062+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:16.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:16.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:17.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:17.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:17.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:17.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:18.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:18.062+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:18.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:18.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:19.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:19.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:19.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:19.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:20.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:20.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:20.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:20.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:21.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:21.022+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:21.824 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:21.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:22.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:22.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:22.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:22.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:22.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:22.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:23.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:23.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:23.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:23.954+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:24.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:24.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:24.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:24.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:25.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:25.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:26.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:26.010+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:26.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:26.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:26.966 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:26.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:27.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:27.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:27.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:27.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:28.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:28.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:28.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:28.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:29.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:29.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:29.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:29.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:30.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:30.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:30.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:30.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:31.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:31.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:31.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:31.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:32.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:32.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:32.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:32.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:33.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:33.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:33.933 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:33.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:34.684 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:34.682+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:34.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:34.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:35.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:35.642+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:35.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:35.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:36.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:36.613+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:36.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:36.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:37.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:37.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:37.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:37.909+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:38.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:38.593+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:38.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:38.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:39.557 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:39.553+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:39.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:39.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:40.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:40.509+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:40.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:40.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:41.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:41.465+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:41.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:41.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:42.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:42.437+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:42.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:42.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:43.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:43.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:43.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:43.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:44.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:44.345+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:44.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:44.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:45.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:45.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:45.982 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:45.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:46.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:46.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:47.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:47.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:47.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:47.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:48.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:48.021+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:48.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:48.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:49.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:49.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:49.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:49.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:50.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:50.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:50.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:50.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:50.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:50.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:51.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:51.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:51.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:51.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:52.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:52.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:52.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:52.957+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:53.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:53.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:53.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:53.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:54.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:54.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:54.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:54.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:55.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:55.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:55.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:55.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:56.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:56.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:56.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:56.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:57.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:57.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:57.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:57.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:58.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:58.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:58.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:58.853+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:59.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:31:59.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:31:59.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:31:59.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:00.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:00.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:00.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:00.905+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:01.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:01.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:01.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:01.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:02.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:02.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:02.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:02.961+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:03.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:03.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:04.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:04.005+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:04.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:04.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:05.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:05.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:05.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:05.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:06.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:06.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:06.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:06.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:07.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:07.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:07.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:07.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:07.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:07.993+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:08.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:08.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:09.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:09.001+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:09.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:09.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:09.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:09.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:10.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:10.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:11.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:11.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:11.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:11.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:12.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:12.053+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:12.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:12.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:13.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:13.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:13.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:13.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:14.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:14.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:14.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:14.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:15.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:15.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:15.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:15.361+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:16.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:16.081+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:16.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:16.393+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:17.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:17.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:17.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:17.405+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:18.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:18.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:18.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:18.437+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:19.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:19.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:19.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:19.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:20.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:20.189+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:20.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:20.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:21.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:21.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:21.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:21.445+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:22.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:22.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:22.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:22.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:23.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:23.233+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:23.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:23.421+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:24.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:24.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:24.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:24.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:25.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:25.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:25.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:25.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:26.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:26.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:26.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:26.417+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:27.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:27.141+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:27.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:27.469+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:28.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:28.192+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:28.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:28.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:29.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:29.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:29.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:29.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:30.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:30.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:30.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:30.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:31.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:31.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:31.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:31.424+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:32.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:32.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:32.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:32.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:33.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:33.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:33.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:33.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:34.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:34.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:34.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:34.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:35.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:35.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:35.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:35.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:36.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:36.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:36.371 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:36.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:37.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:37.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:37.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:37.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:38.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:38.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:38.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:38.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:39.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:39.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:39.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:39.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:40.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:40.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:40.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:40.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:41.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:41.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:41.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:41.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:42.340 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:42.336+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:42.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:42.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:43.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:43.360+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:43.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:43.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:44.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:44.376+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:44.512 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:44.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:45.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:45.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:45.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:45.552+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:46.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:46.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:46.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:46.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:47.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:47.356+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:47.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:47.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:48.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:48.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:48.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:48.520+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:49.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:49.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:49.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:49.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:50.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:50.340+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:50.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:50.528+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:51.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:51.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:51.502 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:51.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:52.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:52.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:52.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:52.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:53.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:53.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:53.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:53.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:54.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:54.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:54.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:54.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:55.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:55.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:55.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:55.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:56.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:56.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:56.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:56.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:57.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:57.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:57.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:57.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:58.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:58.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:58.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:58.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:59.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:32:59.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:32:59.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:32:59.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:00.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:00.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:00.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:00.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:01.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:01.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:01.546 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:01.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:02.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:02.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:02.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:02.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:03.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:03.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:03.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:03.576+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:04.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:04.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:04.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:04.616+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:05.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:05.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:05.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:05.668+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:06.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:06.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:06.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:06.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:07.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:07.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:07.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:07.680+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:08.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:08.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:08.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:08.664+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:09.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:09.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:09.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:09.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:10.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:10.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:10.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:10.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:11.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:11.192+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:11.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:11.692+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:12.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:12.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:12.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:12.672+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:13.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:13.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:13.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:13.644+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:14.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:14.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:14.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:14.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:15.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:15.124+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:15.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:15.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:16.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:16.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:16.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:16.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:17.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:17.148+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:17.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:17.720+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:18.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:18.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:18.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:18.696+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:19.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:19.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:19.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:19.664+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:20.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:20.120+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:20.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:20.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:21.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:21.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:21.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:21.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:22.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:22.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:22.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:22.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:23.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:23.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:23.682 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:23.679+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:24.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:24.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:24.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:24.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:25.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:25.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:25.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:25.683+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:26.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:26.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:26.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:26.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:27.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:27.127+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:27.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:27.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:28.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:28.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:28.684 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:28.683+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:29.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:29.095+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:29.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:29.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:30.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:30.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:30.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:30.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:31.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:31.027+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:31.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:31.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:32.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:32.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:32.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:32.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:32.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:32.979+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:33.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:33.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:33.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:33.979+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:34.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:34.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:34.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:34.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:35.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:35.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:35.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:35.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:36.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:36.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:36.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:36.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:37.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:37.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:37.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:37.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:38.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:38.767+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:38.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:38.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:39.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:39.791+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:39.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:39.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:40.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:40.827+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:40.907 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:40.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:41.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:41.807+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:41.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:41.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:42.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:42.771+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:42.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:42.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:43.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:43.767+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:43.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:43.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:44.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:44.775+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:44.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:44.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:45.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:45.747+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:45.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:45.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:46.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:46.795+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:46.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:46.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:47.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:47.783+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:47.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:47.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:48.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:48.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:48.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:48.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:49.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:49.791+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:49.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:49.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:50.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:50.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:50.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:50.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:51.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:51.807+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:51.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:51.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:52.808 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:52.807+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:52.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:52.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:53.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:53.783+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:53.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:53.891+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:54.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:54.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:54.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:54.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:55.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:55.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:55.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:55.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:56.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:56.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:56.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:56.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:57.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:57.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:57.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:57.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:58.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:58.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:58.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:58.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:59.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:33:59.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:33:59.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:33:59.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:00.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:00.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:00.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:00.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:01.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:01.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:01.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:01.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:02.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:02.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:02.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:02.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:03.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:03.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:03.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:03.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:04.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:04.603+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:04.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:04.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:05.623 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:05.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:05.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:05.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:06.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:06.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:06.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:06.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:07.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:07.619+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:07.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:07.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:08.618 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:08.615+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:08.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:08.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:09.635 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:09.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:09.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:09.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:10.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:10.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:10.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:10.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:11.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:11.647+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:11.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:11.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:12.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:12.611+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:12.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:12.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:13.635 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:13.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:13.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:13.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:14.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:14.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:14.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:14.830+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:15.612 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:15.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:15.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:15.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:16.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:16.598+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:16.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:16.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:17.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:17.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:17.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:17.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:18.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:18.574+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:18.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:18.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:19.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:19.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:19.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:19.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:20.573 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:20.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:20.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:20.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:21.557 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:21.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:21.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:21.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:22.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:22.566+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:22.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:22.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:23.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:23.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:23.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:23.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:24.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:24.510+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:24.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:24.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:25.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:25.530+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:25.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:25.774+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:26.542 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:26.538+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:26.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:26.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:27.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:27.574+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:27.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:27.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:28.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:28.590+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:28.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:28.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:29.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:29.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:29.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:29.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:30.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:30.574+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:30.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:30.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:31.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:31.546+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:31.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:31.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:32.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:32.586+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:32.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:32.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:33.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:33.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:33.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:33.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:34.668 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:34.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:34.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:34.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:35.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:35.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:35.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:35.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:36.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:36.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:36.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:36.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:37.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:37.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:37.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:37.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:38.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:38.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:38.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:38.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:39.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:39.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:39.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:39.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:40.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:40.586+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:40.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:40.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:41.613 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:41.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:41.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:41.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:42.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:42.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:42.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:42.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:43.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:43.642+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:43.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:43.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:44.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:44.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:44.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:44.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:45.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:45.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:45.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:45.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:46.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:46.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:46.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:46.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:47.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:47.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:47.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:47.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:48.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:48.766+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:48.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:48.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:49.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:49.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:49.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:49.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:50.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:50.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:50.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:50.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:51.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:51.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:51.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:51.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:52.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:52.678+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:52.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:52.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:53.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:53.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:53.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:53.782+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:54.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:54.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:54.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:54.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:55.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:55.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:55.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:55.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:56.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:56.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:56.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:56.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:57.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:57.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:57.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:57.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:58.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:58.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:58.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:58.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:59.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:34:59.734+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:34:59.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:34:59.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:00.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:00.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:00.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:00.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:01.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:01.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:01.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:01.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:02.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:02.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:02.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:02.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:03.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:03.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:03.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:03.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:04.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:04.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:04.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:04.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:05.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:05.653+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:05.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:05.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:06.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:06.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:06.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:06.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:07.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:07.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:07.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:07.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:08.606 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:08.601+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:08.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:08.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:09.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:09.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:09.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:09.845+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:10.682 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:10.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:10.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:10.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:11.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:11.673+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:11.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:11.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:12.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:12.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:12.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:12.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:13.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:13.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:13.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:13.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:14.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:14.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:14.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:14.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:15.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:15.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:15.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:15.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:16.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:16.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:16.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:16.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:17.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:17.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:17.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:17.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:18.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:18.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:18.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:18.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:19.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:19.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:19.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:19.901+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:20.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:20.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:20.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:20.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:21.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:21.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:21.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:21.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:22.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:22.673+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:22.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:22.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:23.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:23.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:23.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:23.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:24.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:24.621+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:24.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:24.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:25.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:25.577+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:25.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:25.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:26.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:26.625+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:26.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:26.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:27.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:27.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:27.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:27.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:28.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:28.697+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:28.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:28.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:29.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:29.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:29.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:29.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:30.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:30.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:30.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:30.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:31.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:31.685+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:31.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:31.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:32.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:32.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:32.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:32.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:33.668 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:33.665+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:33.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:33.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:34.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:34.649+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:34.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:34.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:35.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:35.689+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:35.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:35.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:36.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:36.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:36.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:36.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:37.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:37.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:37.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:37.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:38.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:38.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:38.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:38.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:39.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:39.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:39.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:39.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:40.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:40.797+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:40.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:40.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:41.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:41.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:41.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:41.853+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:42.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:42.849+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:42.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:42.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:43.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:43.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:43.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:43.853+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:44.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:44.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:44.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:44.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:45.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:45.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:45.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:45.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:46.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:46.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:46.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:46.881+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:47.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:47.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:47.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:47.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:48.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:48.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:48.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:48.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:49.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:49.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:49.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:49.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:50.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:50.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:50.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:50.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:51.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:51.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:51.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:51.845+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:52.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:52.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:52.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:52.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:53.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:53.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:53.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:53.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:54.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:54.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:54.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:54.849+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:55.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:55.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:55.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:55.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:56.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:56.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:56.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:56.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:57.682 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:57.676+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:57.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:57.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:58.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:58.636+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:58.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:58.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:59.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:35:59.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:35:59.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:35:59.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:00.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:00.608+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:00.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:00.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:01.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:01.632+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:01.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:01.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:02.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:02.624+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:02.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:02.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:03.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:03.668+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:03.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:03.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:04.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:04.652+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:04.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:04.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:05.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:05.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:05.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:05.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:06.660 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:06.656+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:06.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:06.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:07.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:07.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:07.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:07.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:08.617 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:08.612+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:08.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:08.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:09.572 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:09.568+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:09.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:09.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:10.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:10.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:10.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:10.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:11.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:11.536+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:11.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:11.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:12.512 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:12.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:12.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:12.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:13.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:13.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:13.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:13.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:14.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:14.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:14.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:14.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:15.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:15.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:15.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:15.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:16.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:16.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:16.718 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:16.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:17.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:17.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:17.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:17.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:18.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:18.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:18.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:18.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:19.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:19.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:19.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:19.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:20.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:20.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:20.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:20.676+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:21.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:21.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:21.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:21.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:22.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:22.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:22.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:22.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:23.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:23.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:23.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:23.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:24.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:24.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:24.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:24.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:25.455 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:25.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:25.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:25.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:26.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:26.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:26.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:26.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:27.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:27.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:27.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:27.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:28.476 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:28.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:28.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:28.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:29.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:29.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:29.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:29.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:30.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:30.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:30.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:30.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:31.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:31.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:31.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:31.668+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:32.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:32.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:32.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:32.644+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:33.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:33.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:33.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:33.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:34.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:34.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:34.694 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:34.688+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:35.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:35.424+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:35.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:35.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:36.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:36.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:36.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:36.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:37.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:37.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:37.640 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:37.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:38.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:38.552+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:38.672 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:38.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:39.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:39.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:39.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:39.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:40.557 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:40.556+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:40.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:40.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:41.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:41.600+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:41.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:41.688+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:42.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:42.572+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:42.641 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:42.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:43.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:43.584+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:43.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:43.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:44.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:44.600+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:44.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:44.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:45.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:45.580+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:45.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:45.732+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:46.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:46.628+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:46.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:46.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:47.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:47.644+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:47.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:47.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:48.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:48.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:48.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:48.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:49.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:49.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:49.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:49.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:50.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:50.679+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:50.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:50.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:51.635 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:51.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:51.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:51.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:52.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:52.631+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:52.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:52.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:53.593 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:53.591+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:53.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:53.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:54.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:54.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:54.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:54.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:55.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:55.583+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:55.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:55.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:56.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:56.555+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:56.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:56.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:57.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:57.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:57.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:57.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:58.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:58.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:58.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:58.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:59.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:36:59.631+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:36:59.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:36:59.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:00.623 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:00.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:00.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:00.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:01.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:01.591+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:01.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:01.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:02.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:02.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:02.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:02.723+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:03.588 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:03.587+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:03.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:03.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:04.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:04.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:04.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:04.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:05.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:05.647+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:05.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:05.655+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:06.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:06.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:06.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:06.671+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:07.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:07.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:07.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:07.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:08.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:08.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:08.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:08.691+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:09.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:09.647+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:09.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:09.659+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:10.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:10.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:10.667 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:10.667+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:11.635 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:11.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:11.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:11.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:12.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:12.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:12.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:12.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:13.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:13.619+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:13.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:13.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:14.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:14.587+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:14.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:14.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:15.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:15.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:15.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:15.771+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:16.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:16.583+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:16.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:16.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:17.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:17.539+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:17.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:17.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:18.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:18.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:18.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:18.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:19.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:19.503+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:19.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:19.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:20.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:20.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:20.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:20.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:21.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:21.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:21.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:21.783+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:22.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:22.435+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:22.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:22.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:23.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:23.411+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:23.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:23.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:24.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:24.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:24.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:24.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:25.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:25.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:25.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:25.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:26.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:26.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:26.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:26.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:27.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:27.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:27.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:27.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:28.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:28.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:28.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:28.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:29.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:29.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:29.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:29.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:30.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:30.495+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:30.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:30.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:31.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:31.531+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:31.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:31.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:32.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:32.539+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:32.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:32.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:33.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:33.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:33.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:33.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:34.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:34.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:34.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:34.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:35.656 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:35.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:35.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:35.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:36.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:36.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:36.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:36.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:37.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:37.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:37.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:37.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:38.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:38.759+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:38.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:38.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:39.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:39.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:39.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:39.795+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:40.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:40.767+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:40.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:40.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:41.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:41.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:41.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:41.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:42.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:42.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:42.810 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:42.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:43.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:43.754+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:43.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:43.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:44.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:44.722+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:44.820 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:44.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:45.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:45.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:45.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:45.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:46.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:46.766+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:46.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:46.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:47.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:47.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:47.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:47.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:48.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:48.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:48.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:48.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:49.735 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:49.734+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:49.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:49.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:50.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:50.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:50.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:50.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:51.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:51.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:51.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:51.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:52.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:52.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:52.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:52.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:53.802 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:53.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:53.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:53.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:54.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:54.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:54.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:54.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:55.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:55.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:55.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:55.970+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:56.854 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:56.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:56.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:56.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:57.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:57.890+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:57.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:57.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:58.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:58.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:58.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:58.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:59.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:37:59.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:37:59.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:37:59.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:00.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:00.898+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:01.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:01.026+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:01.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:01.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:02.060 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:02.058+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:02.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:02.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:03.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:03.106+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:03.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:03.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:04.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:04.086+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:04.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:04.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:05.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:05.054+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:05.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:05.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:06.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:06.010+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:06.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:06.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:07.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:07.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:07.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:07.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:08.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:08.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:08.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:08.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:08.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:08.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:09.825 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:09.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:09.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:09.982+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:10.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:10.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:10.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:10.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:11.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:11.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:12.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:12.006+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:12.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:12.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:13.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:13.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:13.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:13.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:14.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:14.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:14.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:14.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:15.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:15.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:15.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:15.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:16.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:16.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:16.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:16.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:17.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:17.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:17.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:17.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:18.032 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:18.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:18.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:18.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:19.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:19.058+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:19.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:19.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:20.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:20.102+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:20.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:20.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:21.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:21.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:21.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:21.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:22.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:22.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:22.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:22.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:23.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:23.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:23.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:23.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:24.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:24.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:24.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:24.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:25.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:25.106+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:25.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:25.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:26.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:26.138+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:26.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:26.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:27.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:27.106+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:27.866 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:27.862+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:28.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:28.070+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:28.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:28.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:29.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:29.050+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:29.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:29.890+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:30.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:30.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:30.879 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:30.878+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:31.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:31.006+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:31.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:31.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:32.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:32.042+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:32.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:32.886+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:33.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:33.050+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:33.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:33.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:34.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:34.097+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:34.810 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:34.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:35.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:35.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:35.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:35.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:36.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:36.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:36.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:36.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:37.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:37.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:37.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:37.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:38.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:38.105+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:38.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:38.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:39.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:39.069+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:39.802 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:39.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:40.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:40.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:40.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:40.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:41.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:41.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:41.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:41.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:42.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:42.057+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:42.752 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:42.749+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:43.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:43.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:43.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:43.753+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:44.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:44.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:44.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:44.761+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:45.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:45.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:45.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:45.797+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:46.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:46.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:46.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:46.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:47.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:47.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:47.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:47.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:48.128 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:48.125+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:48.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:48.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:49.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:49.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:49.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:49.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:50.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:50.093+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:50.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:50.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:51.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:51.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:51.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:51.729+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:52.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:52.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:52.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:52.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:53.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:53.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:53.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:53.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:54.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:54.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:54.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:54.693+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:55.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:55.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:55.663 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:55.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:56.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:56.165+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:56.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:56.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:57.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:57.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:57.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:57.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:58.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:58.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:58.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:58.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:59.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:38:59.205+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:38:59.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:38:59.709+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:00.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:00.249+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:00.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:00.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:01.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:01.209+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:01.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:01.749+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:02.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:02.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:02.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:02.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:03.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:03.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:03.752 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:03.749+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:04.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:04.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:04.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:04.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:05.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:05.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:05.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:05.753+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:06.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:06.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:06.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:06.797+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:07.126 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:07.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:07.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:07.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:08.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:08.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:08.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:08.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:09.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:09.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:09.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:09.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:10.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:10.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:10.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:10.929+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:11.110 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:11.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:11.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:11.973+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:12.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:12.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:13.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:13.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:13.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:13.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:14.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:14.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:14.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:14.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:15.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:15.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:15.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:15.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:16.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:16.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:16.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:16.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:17.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:17.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:17.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:17.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:18.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:18.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:18.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:18.125+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:19.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:19.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:19.176 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:19.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:20.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:20.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:20.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:20.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:21.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:21.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:21.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:21.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:22.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:22.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:22.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:22.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:23.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:23.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:23.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:23.165+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:24.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:24.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:24.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:24.157+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:25.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:25.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:25.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:25.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:26.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:26.036+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:26.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:26.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:27.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:26.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:27.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:27.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:28.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:28.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:28.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:28.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:28.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:28.984+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:29.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:29.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:29.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:29.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:30.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:30.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:30.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:30.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:31.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:31.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:32.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:31.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:32.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:32.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:33.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:33.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:33.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:33.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:33.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:33.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:34.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:34.172+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:34.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:34.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:35.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:35.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:35.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:35.964+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:36.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:36.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:36.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:36.940+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:37.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:37.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:37.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:37.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:38.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:38.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:38.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:38.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:39.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:39.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:39.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:39.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:40.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:40.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:41.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:41.016+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:41.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:41.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:42.059 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:42.056+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:42.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:42.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:43.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:43.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:43.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:43.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:44.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:44.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:44.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:44.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:45.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:45.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:45.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:45.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:45.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:45.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:46.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:46.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:46.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:46.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:47.303 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:47.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:47.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:47.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:48.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:48.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:48.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:48.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:49.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:49.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:49.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:49.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:50.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:50.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:50.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:50.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:51.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:51.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:51.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:51.920+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:52.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:52.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:52.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:52.928+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:53.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:53.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:53.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:53.900+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:54.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:54.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:54.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:54.912+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:55.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:55.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:55.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:55.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:56.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:56.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:56.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:56.936+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:57.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:57.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:57.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:57.900+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:58.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:58.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:58.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:58.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:59.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:39:59.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:39:59.854 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:39:59.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:00.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:00.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:00.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:00.884+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:01.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:01.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:01.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:01.904+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:02.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:02.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:02.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:02.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:03.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:03.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:03.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:03.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:04.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:04.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:04.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:04.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:05.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:05.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:05.900 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:05.900+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:06.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:06.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:06.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:06.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:07.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:07.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:07.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:07.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:08.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:08.320+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:09.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:09.016+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:09.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:09.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:09.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:09.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:10.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:10.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:10.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:10.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:11.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:11.360+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:11.962 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:11.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:12.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:12.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:13.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:13.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:13.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:13.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:13.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:13.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:14.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:14.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:14.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:14.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:15.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:15.428+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:16.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:16.012+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:16.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:16.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:16.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:16.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:17.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:17.384+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:17.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:17.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:18.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:18.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:19.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:19.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:19.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:19.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:20.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:20.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:20.398 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:20.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:21.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:21.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:21.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:21.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:21.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:21.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:22.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:22.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:22.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:22.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:23.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:23.355+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:24.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:24.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:24.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:24.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:24.968 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:24.967+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:25.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:25.347+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:26.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:26.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:26.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:26.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:26.968 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:26.967+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:27.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:27.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:27.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:27.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:28.449 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:28.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:28.892 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:28.891+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:29.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:29.487+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:29.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:29.911+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:30.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:30.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:30.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:30.895+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:31.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:31.499+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:31.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:31.903+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:32.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:32.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:32.941 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:32.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:33.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:33.483+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:33.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:33.935+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:34.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:34.467+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:34.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:34.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:35.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:35.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:35.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:35.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:36.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:36.487+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:36.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:36.935+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:37.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:37.487+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:37.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:37.927+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:38.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:38.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:38.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:38.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:39.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:39.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:39.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:39.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:40.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:40.491+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:40.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:40.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:41.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:41.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:41.938 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:41.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:42.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:42.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:42.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:42.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:43.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:43.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:43.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:43.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:44.598 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:44.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:44.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:44.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:45.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:45.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:45.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:45.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:46.603 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:46.603+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:46.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:46.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:47.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:47.595+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:48.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:48.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:48.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:48.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:49.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:49.063+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:49.592 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:49.591+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:50.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:50.031+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:50.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:50.579+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:51.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:51.031+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:51.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:51.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:52.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:51.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:52.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:52.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:52.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:52.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:53.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:53.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:53.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:53.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:54.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:54.583+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:54.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:54.927+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:55.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:55.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:55.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:55.919+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:56.587 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:56.587+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:56.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:56.927+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:57.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:57.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:57.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:57.907+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:58.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:58.531+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:58.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:58.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:59.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:40:59.579+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:40:59.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:40:59.915+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:00.563 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:00.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:00.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:00.879+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:01.605 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:01.603+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:01.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:01.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:02.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:02.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:02.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:02.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:03.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:03.567+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:03.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:03.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:04.598 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:04.595+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:04.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:04.967+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:05.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:05.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:05.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:05.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:06.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:06.631+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:06.962 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:06.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:07.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:07.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:07.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:07.943+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:08.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:08.627+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:08.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:08.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:09.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:09.663+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:09.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:09.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:10.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:10.686+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:10.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:10.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:11.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:11.646+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:11.964 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:11.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:12.606 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:12.602+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:12.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:12.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:13.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:13.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:13.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:13.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:14.595 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:14.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:14.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:14.898+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:15.633 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:15.630+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:15.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:15.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:16.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:16.610+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:16.914 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:16.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:17.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:17.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:17.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:17.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:18.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:18.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:18.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:18.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:19.601 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:19.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:19.952 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:19.950+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:20.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:20.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:20.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:20.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:21.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:21.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:21.895 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:21.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:22.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:22.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:22.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:22.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:23.578 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:23.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:23.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:23.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:24.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:24.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:24.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:24.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:25.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:25.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:25.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:25.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:26.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:26.546+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:26.878 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:26.878+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:27.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:27.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:27.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:27.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:28.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:28.578+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:28.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:28.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:29.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:29.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:29.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:29.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:30.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:30.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:31.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:31.038+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:31.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:31.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:32.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:32.022+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:32.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:32.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:32.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:32.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:33.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:33.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:34.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:34.010+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:34.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:34.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:35.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:35.034+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:35.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:35.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:36.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:36.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:36.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:36.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:37.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:36.998+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:37.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:37.566+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:37.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:37.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:38.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:38.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:39.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:39.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:39.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:39.558+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:40.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:40.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:40.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:40.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:41.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:41.034+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:41.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:41.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:42.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:42.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:42.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:42.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:43.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:43.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:43.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:43.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:43.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:43.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:44.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:44.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:44.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:44.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:45.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:45.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:46.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:46.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:46.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:46.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:46.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:46.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:47.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:47.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:47.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:47.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:48.470 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:48.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:48.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:48.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:49.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:49.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:49.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:49.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:50.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:50.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:50.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:50.998+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:51.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:51.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:51.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:51.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:52.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:52.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:53.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:53.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:53.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:53.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:54.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:54.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:54.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:54.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:55.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:55.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:55.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:55.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:56.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:56.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:56.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:56.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:57.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:57.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:57.470 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:57.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:58.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:58.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:58.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:58.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:59.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:41:59.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:41:59.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:41:59.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:00.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:00.110+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:00.414 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:00.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:01.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:01.110+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:01.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:01.438+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:02.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:02.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:02.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:02.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:03.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:03.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:03.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:03.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:04.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:04.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:04.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:04.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:05.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:05.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:05.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:05.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:06.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:06.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:06.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:06.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:07.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:07.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:07.502 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:07.501+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:08.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:08.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:08.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:08.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:09.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:09.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:09.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:09.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:10.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:10.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:10.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:10.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:11.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:11.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:11.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:11.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:12.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:12.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:12.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:12.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:13.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:13.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:13.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:13.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:14.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:14.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:14.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:14.461+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:15.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:15.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:15.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:15.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:16.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:16.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:16.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:16.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:17.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:17.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:17.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:17.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:18.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:18.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:18.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:18.497+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:19.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:19.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:19.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:19.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:20.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:20.001+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:20.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:20.545+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:21.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:21.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:21.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:21.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:22.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:22.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:22.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:22.613+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:23.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:22.997+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:23.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:23.605+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:23.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:23.953+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:24.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:24.573+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:24.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:24.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:25.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:25.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:25.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:25.985+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:26.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:26.533+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:27.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:27.001+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:27.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:27.509+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:27.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:27.957+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:28.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:28.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:28.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:28.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:29.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:29.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:30.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:30.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:30.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:30.585+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:31.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:31.001+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:31.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:31.601+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:32.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:32.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:32.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:32.565+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:33.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:33.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:33.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:33.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:34.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:34.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:34.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:34.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:35.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:35.005+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:35.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:35.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:36.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:36.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:36.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:36.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:36.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:36.985+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:37.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:37.677+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:37.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:37.937+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:38.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:38.681+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:38.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:38.929+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:39.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:39.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:39.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:39.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:40.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:40.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:40.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:40.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:41.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:41.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:41.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:41.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:42.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:42.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:43.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:43.021+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:43.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:43.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:44.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:44.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:44.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:44.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:45.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:45.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:45.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:45.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:46.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:46.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:46.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:46.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:47.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:47.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:47.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:47.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:48.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:48.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:48.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:48.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:49.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:49.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:49.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:49.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:50.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:50.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:50.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:50.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:51.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:51.129+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:51.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:51.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:52.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:52.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:52.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:52.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:53.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:53.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:53.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:53.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:54.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:54.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:54.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:54.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:55.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:55.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:55.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:55.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:56.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:56.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:56.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:56.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:57.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:57.204+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:57.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:57.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:58.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:58.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:58.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:58.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:59.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:42:59.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:42:59.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:42:59.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:00.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:00.148+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:00.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:00.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:01.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:01.156+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:01.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:01.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:02.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:02.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:02.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:02.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:03.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:03.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:03.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:03.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:04.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:04.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:04.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:04.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:05.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:05.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:05.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:05.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:06.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:06.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:06.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:06.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:07.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:07.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:07.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:07.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:08.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:08.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:08.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:08.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:09.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:09.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:09.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:09.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:10.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:10.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:10.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:10.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:11.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:11.148+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:11.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:11.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:12.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:12.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:12.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:12.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:13.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:13.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:13.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:13.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:14.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:14.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:14.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:14.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:15.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:15.212+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:15.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:15.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:16.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:16.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:16.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:16.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:17.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:17.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:17.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:17.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:18.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:18.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:18.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:18.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:19.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:19.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:19.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:19.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:20.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:20.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:20.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:20.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:21.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:21.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:21.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:21.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:22.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:22.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:22.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:22.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:23.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:23.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:23.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:23.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:24.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:24.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:24.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:24.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:25.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:25.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:25.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:25.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:26.350 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:26.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:26.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:26.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:27.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:27.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:27.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:27.696+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:28.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:28.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:28.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:28.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:29.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:29.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:29.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:29.628+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:30.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:30.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:30.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:30.600+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:31.371 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:31.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:31.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:31.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:32.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:32.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:32.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:32.560+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:33.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:33.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:33.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:33.584+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:34.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:34.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:34.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:34.576+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:35.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:35.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:35.543 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:35.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:36.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:36.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:36.543 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:36.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:37.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:37.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:37.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:37.516+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:38.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:38.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:38.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:38.560+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:39.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:39.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:39.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:39.580+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:40.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:40.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:40.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:40.608+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:41.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:41.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:41.580 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:41.580+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:42.506 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:42.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:42.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:42.556+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:43.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:43.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:43.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:43.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:44.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:44.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:44.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:44.520+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:45.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:45.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:45.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:45.472+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:46.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:46.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:46.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:46.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:47.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:47.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:47.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:47.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:48.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:48.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:48.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:48.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:49.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:49.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:49.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:49.567+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:50.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:50.371+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:50.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:50.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:51.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:51.359+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:51.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:51.659+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:52.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:52.359+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:52.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:52.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:53.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:53.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:53.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:53.655+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:54.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:54.315+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:54.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:54.623+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:55.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:55.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:55.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:55.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:56.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:56.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:56.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:56.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:57.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:57.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:57.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:57.635+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:58.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:58.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:58.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:58.627+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:59.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:43:59.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:43:59.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:43:59.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:00.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:00.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:00.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:00.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:01.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:01.383+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:01.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:01.623+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:02.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:02.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:02.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:02.631+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:03.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:03.347+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:03.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:03.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:04.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:04.359+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:04.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:04.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:05.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:05.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:05.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:05.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:06.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:06.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:06.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:06.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:07.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:07.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:07.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:07.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:08.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:08.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:08.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:08.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:09.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:09.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:09.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:09.491+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:10.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:10.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:10.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:10.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:11.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:11.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:11.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:11.455+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:12.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:12.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:12.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:12.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:13.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:13.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:13.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:13.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:14.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:14.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:14.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:14.475+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:15.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:15.415+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:15.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:15.451+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:16.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:16.411+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:16.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:16.467+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:17.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:17.455+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:17.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:17.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:18.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:18.487+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:18.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:18.491+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:19.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:19.515+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:19.521 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:19.519+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:20.524 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:20.523+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:20.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:20.531+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:21.565 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:21.563+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:21.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:21.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:22.595 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:22.595+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:22.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:22.615+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:23.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:23.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:23.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:23.631+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:24.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:24.603+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:24.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:24.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:25.568 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:25.567+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:25.578 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:25.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:26.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:26.547+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:26.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:26.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:27.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:27.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:27.602 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:27.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:28.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:28.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:28.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:28.555+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:29.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:29.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:29.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:29.583+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:30.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:30.531+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:30.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:30.623+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:31.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:31.563+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:31.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:31.587+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:32.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:32.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:32.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:32.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:33.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:33.555+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:33.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:33.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:34.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:34.531+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:34.565 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:34.563+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:35.555 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:35.555+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:35.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:35.575+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:36.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:36.543+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:36.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:36.547+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:37.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:37.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:37.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:37.559+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:38.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:38.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:38.571 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:38.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:39.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:39.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:39.587 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:39.586+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:40.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:40.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:40.612 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:40.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:41.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:41.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:41.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:41.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:42.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:42.438+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:42.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:42.682+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:43.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:43.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:43.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:43.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:44.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:44.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:44.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:44.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:45.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:45.394+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:45.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:45.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:46.385 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:46.382+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:46.700 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:46.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:47.421 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:47.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:47.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:47.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:48.381 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:48.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:48.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:48.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:49.354 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:49.350+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:49.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:49.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:50.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:50.374+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:50.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:50.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:51.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:51.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:51.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:51.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:52.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:52.362+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:52.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:52.766+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:53.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:53.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:53.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:53.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:54.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:54.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:54.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:54.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:55.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:55.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:55.815 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:55.814+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:56.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:56.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:56.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:56.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:57.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:57.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:57.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:57.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:58.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:58.386+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:58.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:58.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:59.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:44:59.402+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:44:59.656 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:44:59.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:00.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:00.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:00.702 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:00.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:01.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:01.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:01.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:01.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:02.414 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:02.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:02.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:02.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:03.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:03.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:03.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:03.682+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:04.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:04.362+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:04.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:04.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:05.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:05.350+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:05.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:05.598+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:06.389 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:06.386+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:06.570 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:06.566+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:07.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:07.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:07.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:07.578+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:08.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:08.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:08.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:08.602+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:09.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:09.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:09.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:09.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:10.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:10.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:10.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:10.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:11.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:11.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:11.568 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:11.566+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:12.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:12.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:12.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:12.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:13.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:13.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:13.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:13.490+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:14.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:14.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:14.524 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:14.522+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:15.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:15.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:15.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:15.574+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:16.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:16.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:16.618 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:16.614+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:17.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:17.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:17.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:17.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:18.506 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:18.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:18.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:18.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:19.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:19.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:19.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:19.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:20.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:20.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:20.702 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:20.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:21.493 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:21.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:21.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:21.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:22.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:22.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:22.675 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:22.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:23.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:23.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:23.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:23.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:24.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:24.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:24.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:24.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:25.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:25.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:25.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:25.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:26.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:26.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:26.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:26.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:27.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:27.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:27.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:27.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:28.458 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:28.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:28.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:28.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:29.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:29.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:29.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:29.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:30.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:30.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:30.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:30.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:31.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:31.465+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:31.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:31.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:32.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:32.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:32.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:32.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:33.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:33.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:33.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:33.837+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:34.504 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:34.501+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:34.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:34.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:35.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:35.525+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:35.813 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:35.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:36.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:36.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:36.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:36.777+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:37.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:37.593+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:37.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:37.761+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:38.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:38.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:38.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:38.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:39.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:39.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:39.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:39.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:40.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:40.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:40.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:40.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:41.667 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:41.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:41.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:41.761+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:42.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:42.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:42.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:42.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:43.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:43.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:43.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:43.837+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:44.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:44.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:44.825 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:44.821+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:45.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:45.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:45.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:45.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:46.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:46.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:46.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:46.845+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:47.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:47.701+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:47.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:47.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:48.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:48.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:48.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:48.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:49.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:49.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:49.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:49.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:50.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:50.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:50.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:50.849+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:51.718 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:51.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:51.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:51.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:52.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:52.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:52.824 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:52.821+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:53.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:53.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:53.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:53.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:54.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:54.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:54.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:54.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:55.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:55.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:55.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:55.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:56.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:56.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:56.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:56.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:57.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:57.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:57.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:57.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:58.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:58.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:58.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:58.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:59.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:45:59.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:45:59.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:45:59.965+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:00.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:00.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:01.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:01.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:01.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:01.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:01.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:01.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:02.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:02.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:03.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:03.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:03.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:03.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:04.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:04.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:04.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:04.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:05.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:05.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:05.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:05.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:06.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:06.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:06.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:06.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:07.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:07.013+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:07.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:07.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:08.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:08.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:08.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:08.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:09.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:09.005+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:09.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:09.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:09.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:09.973+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:10.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:10.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:11.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:11.005+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:11.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:11.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:11.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:11.973+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:12.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:12.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:12.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:12.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:13.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:13.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:13.964 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:13.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:14.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:14.845+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:14.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:14.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:15.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:15.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:15.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:15.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:16.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:16.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:16.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:16.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:17.906 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:17.901+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:17.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:17.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:18.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:18.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:18.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:18.977+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:19.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:19.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:19.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:19.965+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:20.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:20.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:20.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:20.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:21.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:21.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:21.933 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:21.929+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:22.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:22.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:22.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:22.932+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:23.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:23.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:23.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:23.932+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:24.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:24.900+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:24.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:24.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:25.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:25.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:26.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:26.000+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:26.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:26.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:26.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:26.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:27.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:27.836+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:27.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:27.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:28.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:28.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:28.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:28.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:29.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:29.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:29.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:29.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:30.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:30.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:30.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:30.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:31.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:31.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:32.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:32.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:32.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:32.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:33.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:33.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:33.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:33.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:34.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:34.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:34.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:34.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:35.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:35.072+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:35.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:35.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:36.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:36.060+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:36.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:36.884+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:37.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:37.052+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:37.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:37.888+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:38.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:38.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:38.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:38.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:39.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:39.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:39.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:39.884+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:40.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:40.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:40.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:40.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:41.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:41.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:41.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:41.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:42.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:42.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:42.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:42.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:43.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:43.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:43.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:43.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:43.964 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:43.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:44.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:44.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:45.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:45.012+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:45.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:45.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:45.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:45.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:46.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:46.768+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:47.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:47.036+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:47.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:47.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:48.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:48.036+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:48.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:48.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:49.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:49.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:49.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:49.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:49.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:49.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:50.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:50.768+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:50.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:50.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:51.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:51.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:51.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:51.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:52.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:52.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:52.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:52.940+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:53.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:53.676+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:53.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:53.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:54.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:54.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:54.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:54.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:55.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:55.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:56.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:55.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:56.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:56.732+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:57.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:57.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:57.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:57.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:58.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:58.072+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:58.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:58.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:59.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:46:59.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:46:59.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:46:59.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:00.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:00.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:00.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:00.696+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:01.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:01.012+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:01.701 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:01.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:02.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:02.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:02.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:02.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:03.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:03.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:03.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:03.768+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:04.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:04.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:04.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:04.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:05.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:05.056+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:05.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:05.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:06.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:06.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:06.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:06.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:07.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:07.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:07.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:07.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:08.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:08.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:08.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:08.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:09.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:09.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:09.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:09.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:10.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:10.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:10.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:10.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:11.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:11.108+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:11.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:11.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:12.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:12.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:12.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:12.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:13.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:13.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:13.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:13.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:14.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:14.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:14.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:14.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:15.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:15.079+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:15.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:15.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:16.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:16.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:16.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:16.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:17.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:17.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:17.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:17.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:18.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:18.067+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:18.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:18.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:19.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:19.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:19.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:19.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:20.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:20.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:20.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:20.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:21.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:21.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:21.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:21.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:22.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:22.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:22.672 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:22.671+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:23.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:23.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:23.684 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:23.683+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:24.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:24.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:24.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:24.655+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:25.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:25.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:25.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:25.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:26.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:26.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:26.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:26.619+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:27.083 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:27.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:27.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:27.611+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:28.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:28.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:28.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:28.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:29.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:29.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:29.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:29.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:30.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:30.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:30.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:30.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:31.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:31.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:31.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:31.583+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:32.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:32.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:32.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:32.627+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:33.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:33.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:33.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:33.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:34.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:34.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:34.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:34.591+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:35.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:35.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:35.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:35.555+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:36.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:36.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:36.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:36.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:37.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:37.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:37.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:37.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:38.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:38.067+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:38.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:38.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:39.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:39.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:39.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:39.483+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:40.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:40.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:40.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:40.527+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:41.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:41.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:41.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:41.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:42.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:42.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:42.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:42.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:43.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:43.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:43.555 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:43.555+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:44.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:44.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:44.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:44.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:45.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:45.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:45.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:45.527+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:46.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:46.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:46.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:46.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:47.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:47.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:47.592 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:47.591+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:48.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:48.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:48.634 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:48.631+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:49.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:49.079+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:49.672 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:49.671+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:50.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:50.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:50.667 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:50.667+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:51.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:51.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:51.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:51.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:52.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:52.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:52.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:52.659+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:53.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:53.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:53.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:53.623+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:54.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:54.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:54.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:54.627+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:55.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:55.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:55.602 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:55.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:56.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:56.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:56.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:56.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:57.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:57.031+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:57.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:57.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:58.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:58.079+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:58.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:58.555+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:59.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:47:59.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:47:59.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:47:59.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:00.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:00.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:00.506 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:00.507+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:01.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:01.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:01.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:01.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:02.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:02.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:02.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:02.467+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:03.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:03.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:03.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:03.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:04.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:04.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:04.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:04.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:04.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:04.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:05.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:05.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:05.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:05.967+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:06.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:06.382+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:06.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:06.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:07.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:07.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:07.952 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:07.950+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:08.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:08.434+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:08.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:08.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:09.466 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:09.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:09.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:09.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:10.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:10.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:10.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:10.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:11.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:11.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:11.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:11.930+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:12.470 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:12.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:12.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:12.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:13.512 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:13.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:13.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:13.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:14.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:14.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:14.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:14.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:15.449 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:15.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:15.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:15.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:16.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:16.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:16.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:16.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:17.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:17.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:17.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:17.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:18.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:18.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:18.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:18.814+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:19.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:19.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:19.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:19.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:20.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:20.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:20.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:20.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:21.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:21.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:21.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:21.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:22.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:22.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:22.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:22.814+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:23.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:23.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:23.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:23.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:24.504 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:24.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:24.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:24.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:25.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:25.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:25.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:25.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:26.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:26.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:26.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:26.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:27.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:27.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:27.835 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:27.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:28.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:28.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:28.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:28.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:29.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:29.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:29.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:29.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:30.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:30.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:30.808 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:30.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:31.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:31.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:31.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:31.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:32.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:32.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:32.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:32.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:33.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:33.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:33.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:33.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:34.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:34.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:34.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:34.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:35.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:35.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:35.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:35.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:36.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:36.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:36.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:36.862+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:37.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:37.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:37.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:37.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:38.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:38.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:38.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:38.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:39.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:39.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:39.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:39.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:40.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:40.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:40.792 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:40.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:41.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:41.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:41.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:41.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:42.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:42.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:42.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:42.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:43.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:43.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:43.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:43.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:44.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:44.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:44.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:44.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:45.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:45.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:45.874 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:45.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:46.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:46.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:46.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:46.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:47.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:47.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:47.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:47.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:48.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:48.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:48.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:48.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:49.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:49.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:49.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:49.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:50.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:50.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:50.808 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:50.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:51.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:51.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:51.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:51.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:52.479 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:52.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:52.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:52.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:53.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:53.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:53.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:53.878+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:54.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:54.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:54.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:54.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:55.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:55.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:55.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:55.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:56.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:56.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:56.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:56.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:57.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:57.434+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:57.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:57.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:58.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:58.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:58.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:58.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:59.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:48:59.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:48:59.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:48:59.977+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:00.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:00.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:00.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:00.973+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:01.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:01.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:02.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:02.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:02.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:02.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:02.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:02.993+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:03.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:03.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:03.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:03.953+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:04.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:04.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:04.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:04.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:05.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:05.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:05.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:05.985+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:06.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:06.557+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:06.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:06.957+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:07.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:07.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:07.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:07.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:08.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:08.525+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:08.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:08.953+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:09.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:09.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:09.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:09.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:10.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:10.517+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:10.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:10.909+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:11.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:11.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:11.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:11.913+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:12.555 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:12.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:12.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:12.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:13.516 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:13.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:13.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:13.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:14.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:14.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:14.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:14.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:15.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:15.545+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:15.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:15.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:16.571 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:16.569+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:16.887 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:16.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:17.595 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:17.593+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:17.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:17.897+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:18.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:18.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:18.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:18.937+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:19.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:19.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:19.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:19.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:20.634 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:20.629+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:20.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:20.905+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:21.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:21.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:21.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:21.905+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:22.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:22.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:22.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:22.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:23.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:23.605+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:23.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:23.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:24.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:24.581+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:24.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:24.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:25.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:25.597+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:25.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:25.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:26.637 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:26.633+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:26.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:26.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:27.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:27.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:27.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:27.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:28.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:28.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:28.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:28.909+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:29.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:29.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:29.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:29.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:30.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:30.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:30.900 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:30.897+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:31.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:31.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:31.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:31.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:32.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:32.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:32.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:32.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:33.712 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:33.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:33.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:33.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:34.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:34.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:34.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:34.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:35.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:35.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:36.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:35.997+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:36.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:36.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:37.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:37.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:37.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:37.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:38.043 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:38.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:38.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:38.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:39.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:39.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:39.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:39.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:40.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:40.005+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:40.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:40.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:41.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:41.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:41.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:41.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:42.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:42.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:42.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:42.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:42.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:42.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:43.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:43.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:43.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:43.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:44.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:44.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:44.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:44.977+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:45.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:45.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:45.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:45.953+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:46.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:46.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:46.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:46.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:47.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:47.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:47.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:47.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:48.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:48.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:48.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:48.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:49.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:49.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:49.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:49.937+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:50.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:50.888+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:50.902 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:50.896+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:51.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:51.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:51.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:51.940+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:52.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:52.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:52.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:52.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:53.892 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:53.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:53.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:53.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:54.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:54.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:54.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:54.928+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:55.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:55.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:55.952 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:55.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:56.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:56.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:56.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:56.908+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:57.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:57.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:57.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:57.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:58.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:58.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:58.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:58.860+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:59.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:49:59.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:49:59.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:49:59.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:00.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:00.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:00.878 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:00.872+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:01.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:01.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:01.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:01.876+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:02.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:02.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:02.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:02.876+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:03.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:03.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:03.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:03.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:04.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:04.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:04.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:04.864+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:05.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:05.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:05.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:05.840+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:06.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:06.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:06.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:06.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:07.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:07.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:07.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:07.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:08.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:08.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:08.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:08.828+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:09.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:09.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:09.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:09.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:10.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:10.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:10.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:10.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:11.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:11.820+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:11.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:11.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:12.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:12.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:12.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:12.864+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:13.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:13.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:13.862 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:13.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:14.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:14.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:14.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:14.820+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:15.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:15.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:15.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:15.828+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:16.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:16.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:16.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:16.860+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:17.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:17.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:17.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:17.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:18.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:18.768+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:18.878 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:18.872+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:19.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:19.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:19.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:19.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:20.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:20.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:20.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:20.920+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:21.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:21.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:21.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:21.940+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:22.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:22.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:22.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:22.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:23.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:23.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:23.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:23.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:24.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:24.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:24.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:24.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:25.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:25.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:25.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:25.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:26.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:26.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:26.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:26.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:27.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:27.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:27.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:27.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:28.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:28.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:28.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:28.928+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:29.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:29.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:29.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:29.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:30.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:30.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:30.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:30.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:31.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:31.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:31.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:31.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:32.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:32.900+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:32.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:32.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:33.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:33.888+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:33.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:33.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:34.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:34.896+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:34.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:34.944+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:35.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:35.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:35.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:35.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:36.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:36.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:37.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:37.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:37.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:37.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:38.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:38.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:38.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:38.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:38.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:38.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:39.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:39.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:39.941 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:39.940+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:40.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:40.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:40.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:40.904+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:41.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:41.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:41.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:41.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:42.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:42.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:42.963 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:42.963+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:43.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:43.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:43.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:43.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:44.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:44.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:44.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:44.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:45.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:45.891+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:46.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:46.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:46.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:46.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:47.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:47.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:47.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:47.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:48.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:48.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:48.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:48.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:49.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:49.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:49.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:49.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:50.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:50.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:50.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:50.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:51.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:51.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:51.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:51.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:52.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:52.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:52.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:52.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:53.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:53.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:53.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:53.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:54.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:54.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:54.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:54.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:55.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:55.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:55.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:55.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:56.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:56.179+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:56.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:56.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:57.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:57.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:57.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:57.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:58.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:58.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:58.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:58.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:59.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:50:59.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:50:59.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:50:59.963+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:00.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:00.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:00.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:00.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:01.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:01.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:01.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:01.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:02.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:02.223+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:02.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:02.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:03.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:03.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:03.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:03.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:04.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:04.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:04.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:04.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:05.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:05.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:05.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:05.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:06.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:06.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:06.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:06.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:07.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:07.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:07.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:07.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:08.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:08.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:08.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:08.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:09.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:09.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:09.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:09.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:10.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:10.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:10.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:10.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:11.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:11.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:11.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:11.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:12.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:12.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:12.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:12.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:13.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:13.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:13.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:13.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:14.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:14.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:14.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:14.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:15.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:15.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:15.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:15.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:16.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:16.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:16.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:16.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:17.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:17.379+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:17.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:17.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:18.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:18.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:18.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:18.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:19.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:19.367+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:19.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:19.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:20.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:20.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:20.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:20.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:21.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:21.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:21.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:21.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:22.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:22.359+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:22.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:22.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:23.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:23.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:23.907 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:23.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:24.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:24.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:24.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:24.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:25.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:25.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:25.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:25.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:26.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:26.439+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:27.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:27.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:27.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:27.415+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:28.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:28.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:28.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:28.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:29.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:29.027+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:29.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:29.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:30.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:30.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:30.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:30.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:31.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:31.027+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:31.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:31.431+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:32.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:32.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:32.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:32.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:32.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:32.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:33.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:33.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:33.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:33.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:34.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:34.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:34.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:34.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:35.426 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:35.423+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:35.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:35.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:36.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:36.402+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:36.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:36.970+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:37.419 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:37.418+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:37.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:37.966+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:38.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:38.466+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:38.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:38.958+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:39.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:39.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:40.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:40.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:40.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:40.474+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:41.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:41.050+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:41.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:41.466+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:42.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:42.058+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:42.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:42.490+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:43.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:43.110+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:43.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:43.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:44.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:44.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:44.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:44.478+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:45.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:45.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:45.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:45.502+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:46.008 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:46.006+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:46.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:46.494+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:47.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:46.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:47.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:47.510+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:48.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:48.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:48.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:48.494+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:49.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:49.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:49.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:49.518+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:49.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:49.982+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:50.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:50.538+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:51.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:51.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:51.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:51.546+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:52.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:52.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:52.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:52.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:53.099 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:53.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:53.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:53.494+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:54.071 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:54.070+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:54.543 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:54.542+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:55.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:55.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:55.540 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:55.538+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:56.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:56.090+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:56.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:56.542+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:57.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:57.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:57.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:57.590+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:58.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:58.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:58.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:58.618+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:59.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:51:59.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:51:59.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:51:59.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:00.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:00.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:00.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:00.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:01.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:01.082+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:01.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:01.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:02.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:02.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:02.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:02.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:03.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:03.138+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:03.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:03.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:04.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:04.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:04.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:04.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:05.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:05.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:05.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:05.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:06.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:06.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:06.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:06.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:07.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:07.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:07.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:07.726+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:08.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:08.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:08.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:08.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:09.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:09.226+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:09.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:09.774+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:10.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:10.186+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:10.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:10.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:11.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:11.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:11.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:11.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:12.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:12.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:12.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:12.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:13.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:13.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:13.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:13.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:14.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:14.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:14.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:14.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:15.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:15.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:15.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:15.750+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:16.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:16.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:16.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:16.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:17.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:17.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:17.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:17.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:18.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:18.262+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:18.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:18.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:19.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:19.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:19.802 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:19.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:20.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:20.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:20.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:20.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:21.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:21.334+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:21.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:21.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:22.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:22.338+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:22.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:22.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:23.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:23.318+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:23.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:23.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:24.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:24.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:24.879 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:24.878+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:25.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:25.346+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:25.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:25.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:26.361 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:26.358+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:26.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:26.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:27.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:27.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:27.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:27.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:28.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:28.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:28.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:28.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:29.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:29.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:29.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:29.797+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:30.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:30.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:30.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:30.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:31.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:31.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:31.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:31.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:32.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:32.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:32.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:32.729+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:33.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:33.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:33.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:33.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:34.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:34.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:34.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:34.693+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:35.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:35.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:35.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:35.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:36.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:36.285+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:36.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:36.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:37.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:37.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:37.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:37.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:38.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:38.257+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:38.750 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:38.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:39.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:39.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:39.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:39.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:40.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:40.297+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:40.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:40.753+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:41.274 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:41.269+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:41.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:41.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:42.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:42.309+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:42.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:42.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:43.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:43.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:43.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:43.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:44.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:44.289+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:44.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:44.845+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:45.303 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:45.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:45.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:45.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:46.335 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:46.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:46.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:46.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:47.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:47.373+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:47.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:47.821+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:48.338 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:48.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:48.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:48.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:49.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:49.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:49.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:49.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:50.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:50.297+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:50.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:50.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:51.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:51.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:51.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:51.777+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:52.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:52.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:52.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:52.769+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:53.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:53.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:53.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:53.733+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:54.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:54.313+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:54.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:54.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:55.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:55.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:55.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:55.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:56.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:56.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:56.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:56.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:57.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:57.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:57.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:57.837+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:58.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:58.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:58.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:58.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:59.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:52:59.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:52:59.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:52:59.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:00.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:00.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:00.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:00.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:01.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:01.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:01.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:01.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:02.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:02.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:02.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:02.797+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:03.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:03.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:03.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:03.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:04.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:04.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:04.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:04.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:05.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:05.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:05.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:05.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:06.429 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:06.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:06.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:06.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:07.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:07.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:07.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:07.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:08.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:08.381+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:08.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:08.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:09.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:09.409+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:09.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:09.821+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:10.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:10.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:10.873 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:10.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:11.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:11.373+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:11.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:11.897+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:12.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:12.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:12.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:12.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:13.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:13.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:13.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:13.913+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:14.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:14.285+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:14.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:14.881+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:15.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:15.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:15.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:15.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:16.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:16.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:16.887 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:16.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:17.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:17.381+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:17.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:17.837+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:18.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:18.341+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:18.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:18.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:19.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:19.305+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:19.803 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:19.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:20.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:20.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:20.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:20.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:21.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:21.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:21.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:21.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:22.398 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:22.392+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:22.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:22.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:23.379 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:23.376+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:23.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:23.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:24.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:24.396+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:24.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:24.820+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:25.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:25.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:25.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:25.820+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:26.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:26.320+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:26.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:26.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:27.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:27.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:27.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:27.768+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:28.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:28.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:28.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:28.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:29.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:29.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:29.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:29.808+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:30.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:30.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:30.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:30.836+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:31.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:31.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:31.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:31.836+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:32.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:32.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:32.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:32.828+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:33.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:33.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:33.792 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:33.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:34.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:34.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:34.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:34.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:35.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:35.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:35.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:35.756+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:36.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:36.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:36.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:36.712+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:37.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:37.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:37.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:37.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:38.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:38.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:38.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:38.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:39.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:39.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:39.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:39.700+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:40.274 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:40.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:40.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:40.672+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:41.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:41.304+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:41.684 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:41.680+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:42.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:42.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:42.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:42.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:43.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:43.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:43.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:43.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:44.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:44.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:44.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:44.664+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:45.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:45.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:45.684 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:45.680+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:46.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:46.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:46.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:46.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:47.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:47.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:47.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:47.676+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:48.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:48.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:48.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:48.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:49.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:49.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:49.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:49.756+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:50.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:50.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:50.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:50.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:51.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:51.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:51.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:51.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:52.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:52.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:52.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:52.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:53.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:53.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:53.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:53.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:54.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:54.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:54.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:54.684+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:55.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:55.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:55.735 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:55.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:56.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:56.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:56.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:56.692+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:57.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:57.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:57.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:57.708+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:58.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:58.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:58.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:58.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:59.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:53:59.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:53:59.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:53:59.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:00.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:00.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:00.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:00.744+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:01.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:01.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:01.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:01.772+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:02.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:02.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:02.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:02.824+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:03.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:03.172+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:03.847 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:03.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:04.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:04.172+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:04.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:04.836+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:05.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:05.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:05.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:05.868+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:06.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:06.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:06.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:06.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:07.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:07.192+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:07.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:07.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:08.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:08.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:08.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:08.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:09.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:09.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:09.792 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:09.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:10.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:10.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:10.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:10.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:11.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:11.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:11.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:11.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:12.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:12.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:12.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:12.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:13.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:13.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:13.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:13.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:14.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:14.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:14.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:14.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:15.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:15.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:15.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:15.795+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:16.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:16.375+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:16.847 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:16.847+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:17.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:17.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:17.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:17.887+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:18.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:18.367+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:18.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:18.855+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:19.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:19.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:19.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:19.831+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:20.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:20.307+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:20.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:20.815+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:21.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:21.331+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:21.825 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:21.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:22.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:22.347+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:22.824 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:22.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:23.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:23.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:23.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:23.835+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:24.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:24.343+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:24.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:24.883+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:25.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:25.323+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:25.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:25.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:26.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:26.295+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:26.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:26.927+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:27.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:27.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:27.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:27.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:28.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:28.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:28.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:28.927+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:29.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:29.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:29.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:29.887+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:30.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:30.283+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:30.873 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:30.871+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:31.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:31.283+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:31.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:31.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:32.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:32.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:32.902 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:32.899+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:33.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:33.307+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:33.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:33.923+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:34.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:34.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:34.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:34.963+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:35.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:35.295+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:35.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:35.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:36.321 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:36.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:36.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:36.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:37.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:37.311+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:37.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:37.975+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:38.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:38.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:39.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:39.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:39.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:39.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:40.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:40.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:40.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:40.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:41.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:41.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:41.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:41.279+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:41.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:41.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:42.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:42.323+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:43.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:43.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:43.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:43.351+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:44.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:44.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:44.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:44.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:45.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:45.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:45.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:45.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:46.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:46.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:46.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:46.415+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:46.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:46.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:47.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:47.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:47.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:47.943+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:48.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:48.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:48.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:48.903+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:49.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:49.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:49.952 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:49.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:50.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:50.475+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:50.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:50.967+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:51.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:51.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:51.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:51.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:52.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:52.483+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:52.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:52.911+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:53.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:53.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:53.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:53.875+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:54.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:54.459+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:54.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:54.915+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:55.418 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:55.415+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:55.879 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:55.879+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:56.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:56.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:56.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:56.863+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:57.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:57.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:57.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:57.847+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:58.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:58.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:58.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:58.815+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:59.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:54:59.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:54:59.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:54:59.847+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:00.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:00.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:00.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:00.851+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:01.562 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:01.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:01.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:01.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:02.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:02.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:02.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:02.795+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:03.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:03.491+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:03.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:03.799+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:04.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:04.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:04.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:04.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:05.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:05.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:05.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:05.775+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:06.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:06.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:06.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:06.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:07.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:07.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:07.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:07.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:08.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:08.558+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:08.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:08.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:09.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:09.546+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:09.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:09.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:10.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:10.566+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:10.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:10.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:11.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:11.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:11.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:11.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:12.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:12.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:12.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:12.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:13.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:13.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:13.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:13.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:14.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:14.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:14.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:14.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:15.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:15.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:15.675 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:15.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:16.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:16.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:16.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:16.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:17.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:17.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:17.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:17.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:18.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:18.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:18.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:18.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:19.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:19.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:19.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:19.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:20.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:20.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:20.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:20.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:21.421 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:21.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:21.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:21.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:22.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:22.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:22.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:22.642+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:23.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:23.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:23.612 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:23.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:24.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:24.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:24.649 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:24.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:25.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:25.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:25.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:25.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:26.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:26.402+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:26.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:26.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:27.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:27.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:27.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:27.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:28.470 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:28.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:28.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:28.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:29.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:29.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:29.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:29.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:30.466 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:30.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:30.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:30.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:31.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:31.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:31.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:31.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:32.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:32.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:32.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:32.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:33.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:33.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:33.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:33.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:34.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:34.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:34.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:34.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:35.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:35.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:35.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:35.750+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:36.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:36.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:36.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:36.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:37.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:37.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:37.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:37.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:38.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:38.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:38.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:38.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:39.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:39.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:39.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:39.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:40.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:40.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:40.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:40.814+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:41.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:41.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:41.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:41.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:42.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:42.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:42.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:42.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:43.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:43.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:43.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:43.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:44.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:44.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:44.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:44.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:45.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:45.578+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:45.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:45.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:46.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:46.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:46.866 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:46.862+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:47.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:47.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:47.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:47.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:48.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:48.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:48.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:48.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:49.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:49.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:49.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:49.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:50.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:50.558+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:50.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:50.878+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:51.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:51.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:51.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:51.898+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:52.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:52.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:52.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:52.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:53.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:53.562+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:53.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:53.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:54.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:54.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:54.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:54.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:55.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:55.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:55.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:55.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:56.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:56.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:56.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:56.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:57.422 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:57.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:57.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:57.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:58.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:58.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:58.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:55:58.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:55:59.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:55:59.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:00.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:00.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:00.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:00.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:01.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:01.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:01.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:01.433+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:02.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:02.053+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:02.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:02.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:03.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:03.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:03.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:03.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:04.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:04.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:04.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:04.417+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:05.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:05.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:05.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:05.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:06.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:06.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:06.361 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:06.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:07.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:07.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:07.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:07.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:08.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:08.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:08.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:08.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:09.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:09.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:09.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:09.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:10.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:10.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:10.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:10.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:11.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:11.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:11.379 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:11.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:12.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:12.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:12.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:12.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:13.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:13.169+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:13.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:13.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:14.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:14.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:14.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:14.341+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:15.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:15.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:15.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:15.305+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:16.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:16.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:16.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:16.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:17.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:17.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:17.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:17.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:18.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:18.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:18.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:18.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:19.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:19.005+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:19.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:19.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:20.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:20.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:20.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:20.309+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:21.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:21.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:21.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:21.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:22.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:22.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:22.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:22.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:23.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:23.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:23.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:23.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:24.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:24.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:24.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:24.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:25.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:25.033+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:25.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:25.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:26.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:26.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:26.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:26.285+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:27.017 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:27.013+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:27.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:27.321+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:28.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:28.061+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:28.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:28.353+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:29.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:29.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:29.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:29.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:30.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:30.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:30.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:30.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:31.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:31.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:31.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:31.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:32.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:32.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:32.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:32.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:33.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:33.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:33.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:33.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:34.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:34.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:34.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:34.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:35.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:35.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:35.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:35.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:36.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:36.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:36.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:36.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:37.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:37.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:37.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:37.497+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:38.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:38.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:38.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:38.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:39.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:39.065+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:39.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:39.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:40.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:40.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:40.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:40.545+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:41.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:41.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:41.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:41.569+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:42.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:42.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:42.606 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:42.605+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:43.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:43.073+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:43.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:43.653+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:44.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:44.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:44.682 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:44.677+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:45.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:45.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:45.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:45.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:46.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:46.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:46.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:46.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:47.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:47.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:47.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:47.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:48.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:48.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:48.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:48.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:49.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:49.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:49.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:49.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:50.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:50.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:50.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:50.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:51.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:51.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:51.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:51.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:52.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:52.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:52.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:52.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:53.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:53.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:53.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:53.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:54.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:54.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:54.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:54.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:55.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:55.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:55.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:55.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:56.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:56.360+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:56.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:56.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:57.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:57.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:57.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:57.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:58.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:58.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:58.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:58.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:59.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:56:59.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:56:59.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:56:59.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:00.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:00.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:00.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:00.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:01.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:01.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:01.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:01.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:02.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:02.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:02.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:02.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:03.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:03.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:03.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:03.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:04.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:04.220+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:04.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:04.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:05.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:05.204+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:05.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:05.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:06.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:06.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:06.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:06.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:07.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:07.200+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:07.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:07.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:08.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:08.200+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:08.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:08.732+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:09.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:09.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:09.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:09.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:10.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:10.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:10.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:10.688+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:11.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:11.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:11.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:11.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:12.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:12.248+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:12.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:12.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:13.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:13.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:13.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:13.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:14.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:14.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:14.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:14.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:15.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:15.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:15.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:15.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:16.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:16.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:16.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:16.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:17.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:17.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:17.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:17.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:18.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:18.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:18.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:18.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:19.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:19.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:19.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:19.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:20.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:20.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:20.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:20.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:21.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:21.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:21.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:21.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:22.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:22.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:22.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:22.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:23.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:23.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:23.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:23.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:24.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:24.360+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:24.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:24.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:25.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:25.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:25.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:25.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:26.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:26.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:26.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:26.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:27.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:27.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:27.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:27.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:28.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:28.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:28.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:28.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:29.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:29.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:29.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:29.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:30.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:30.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:30.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:30.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:31.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:31.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:31.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:31.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:32.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:32.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:32.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:32.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:33.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:33.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:33.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:33.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:34.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:34.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:34.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:34.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:35.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:35.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:35.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:35.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:36.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:36.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:36.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:36.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:37.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:37.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:37.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:37.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:38.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:38.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:38.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:38.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:39.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:39.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:39.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:39.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:40.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:40.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:40.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:40.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:41.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:41.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:41.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:41.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:42.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:42.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:42.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:42.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:43.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:43.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:43.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:43.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:44.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:44.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:44.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:44.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:45.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:45.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:45.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:45.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:46.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:46.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:46.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:46.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:47.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:47.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:47.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:47.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:48.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:48.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:48.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:48.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:49.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:49.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:49.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:49.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:50.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:50.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:50.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:50.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:51.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:51.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:51.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:51.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:52.330 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:52.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:52.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:52.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:53.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:53.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:53.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:53.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:54.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:54.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:54.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:54.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:55.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:55.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:55.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:55.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:56.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:56.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:56.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:56.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:57.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:57.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:57.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:57.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:58.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:58.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:58.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:58.795+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:59.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:57:59.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:57:59.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:57:59.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:00.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:00.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:00.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:00.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:01.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:01.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:01.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:01.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:02.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:02.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:02.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:02.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:03.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:03.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:03.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:03.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:04.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:04.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:04.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:04.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:05.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:05.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:05.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:05.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:06.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:06.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:06.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:06.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:07.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:07.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:07.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:07.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:08.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:08.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:08.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:08.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:09.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:09.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:09.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:09.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:10.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:10.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:10.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:10.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:11.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:11.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:11.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:11.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:12.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:12.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:12.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:12.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:13.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:13.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:13.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:13.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:14.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:14.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:14.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:14.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:15.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:15.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:15.905 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:15.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:16.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:16.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:16.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:16.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:17.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:17.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:17.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:17.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:18.349 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:18.347+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:18.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:18.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:19.366 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:19.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:19.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:19.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:20.351 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:20.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:20.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:20.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:21.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:21.383+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:21.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:21.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:22.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:22.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:22.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:22.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:23.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:23.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:23.807 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:23.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:24.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:24.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:24.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:24.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:25.455 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:25.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:25.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:25.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:26.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:26.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:26.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:26.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:27.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:27.459+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:27.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:27.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:28.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:28.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:28.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:28.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:29.468 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:29.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:29.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:29.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:30.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:30.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:30.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:30.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:31.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:31.459+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:31.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:31.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:32.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:32.463+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:32.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:32.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:33.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:33.439+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:34.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:34.015+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:34.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:34.459+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:34.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:34.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:35.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:35.507+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:36.033 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:36.031+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:36.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:36.507+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:37.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:37.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:37.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:37.507+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:38.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:38.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:38.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:38.471+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:39.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:39.047+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:39.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:39.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:40.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:40.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:40.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:40.482+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:41.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:41.054+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:41.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:41.486+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:42.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:42.058+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:42.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:42.502+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:43.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:43.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:43.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:43.454+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:44.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:44.082+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:44.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:44.446+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:45.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:45.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:45.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:45.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:46.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:46.082+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:46.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:46.394+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:47.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:47.050+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:47.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:47.386+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:48.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:48.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:48.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:48.422+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:49.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:49.058+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:49.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:49.470+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:50.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:50.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:50.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:50.474+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:51.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:51.058+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:51.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:51.462+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:52.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:52.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:52.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:52.438+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:53.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:53.050+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:53.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:53.466+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:54.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:54.090+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:54.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:54.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:55.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:55.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:55.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:55.386+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:56.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:56.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:56.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:56.426+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:57.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:57.026+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:57.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:57.406+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:58.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:58.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:58.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:58.362+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:58.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:58.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:59.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:58:59.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:58:59.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:58:59.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:00.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:00.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:01.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:01.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:01.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:01.386+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:02.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:02.022+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:02.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:02.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:03.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:03.074+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:03.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:03.442+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:04.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:04.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:04.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:04.454+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:05.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:05.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:05.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:05.494+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:05.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:05.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:06.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:06.470+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:07.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:07.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:07.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:07.454+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:08.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:08.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:08.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:08.450+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:09.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:09.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:09.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:09.406+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:09.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:09.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:10.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:10.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:10.967 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:10.966+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:11.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:11.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:11.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:11.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:12.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:12.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:12.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:12.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:13.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:13.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:13.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:13.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:14.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:14.386+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:14.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:14.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:15.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:15.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:15.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:15.954+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:16.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:16.406+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:17.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:17.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:17.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:17.414+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:18.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:18.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:18.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:18.434+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:18.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:18.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:19.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:19.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:19.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:19.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:20.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:20.414+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:20.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:20.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:21.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:21.462+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:21.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:21.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:22.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:22.462+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:22.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:22.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:23.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:23.418+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:23.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:23.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:24.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:24.446+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:24.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:24.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:25.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:25.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:25.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:25.942+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:26.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:26.358+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:26.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:26.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:27.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:27.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:27.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:27.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:28.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:28.326+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:28.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:28.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:29.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:29.330+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:29.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:29.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:30.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:30.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:31.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:31.006+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:31.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:31.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:32.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:32.042+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:32.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:32.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:33.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:33.057+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:33.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:33.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:34.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:34.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:34.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:34.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:35.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:35.105+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:35.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:35.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:36.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:36.093+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:36.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:36.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:37.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:37.125+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:37.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:37.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:38.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:38.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:38.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:38.397+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:39.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:39.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:39.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:39.381+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:40.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:40.013+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:40.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:40.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:41.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:41.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:41.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:41.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:42.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:42.081+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:42.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:42.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:43.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:43.049+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:43.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:43.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:44.094 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:44.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:44.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:44.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:45.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:45.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:45.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:45.509+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:46.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:46.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:46.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:46.469+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:47.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:47.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:47.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:47.465+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:48.134 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:48.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:48.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:48.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:49.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:49.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:49.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:49.397+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:50.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:50.093+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:50.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:50.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:51.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:51.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:51.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:51.325+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:52.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:52.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:52.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:52.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:53.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:53.069+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:53.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:53.397+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:54.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:54.053+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:54.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:54.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:55.016 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:55.013+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:55.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:55.397+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:56.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:56.053+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:56.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:56.405+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:57.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:57.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:57.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:57.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:58.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:58.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:58.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:58.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:59.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-11T23:59:59.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-11T23:59:59.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-11T23:59:59.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:00.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:00.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:00.466 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:00.465+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:01.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:01.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:01.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:01.421+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:02.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:02.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:02.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:02.385+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:03.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:03.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:03.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:03.341+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:04.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:04.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:04.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:04.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:05.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:05.117+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:05.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:05.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:06.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:06.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:06.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:06.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:07.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:07.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:07.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:07.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:08.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:08.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:08.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:08.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:09.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:09.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:09.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:09.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:10.137 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:10.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:10.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:10.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:11.179 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:11.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:11.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:11.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:12.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:12.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:12.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:12.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:13.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:13.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:13.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:13.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:14.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:14.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:14.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:14.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:15.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:15.209+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:15.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:15.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:16.176 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:16.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:16.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:16.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:17.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:17.189+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:17.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:17.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:18.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:18.233+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:18.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:18.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:19.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:19.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:19.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:19.277+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:20.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:20.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:20.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:20.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:21.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:21.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:21.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:21.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:22.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:22.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:22.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:22.341+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:23.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:23.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:23.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:23.353+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:24.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:24.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:24.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:24.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:25.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:25.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:25.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:25.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:26.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:26.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:26.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:26.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:27.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:27.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:27.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:27.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:28.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:28.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:28.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:28.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:29.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:29.372+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:29.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:29.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:30.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:30.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:30.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:30.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:31.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:31.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:31.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:31.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:32.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:32.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:32.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:32.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:33.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:33.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:33.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:33.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:34.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:34.396+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:34.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:34.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:35.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:35.356+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:35.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:35.424+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:36.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:36.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:36.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:36.408+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:37.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:37.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:37.366 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:37.360+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:38.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:38.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:38.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:38.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:39.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:39.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:39.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:39.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:40.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:40.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:40.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:40.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:41.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:41.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:41.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:41.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:42.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:42.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:42.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:42.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:43.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:43.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:43.351 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:43.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:44.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:44.172+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:44.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:44.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:45.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:45.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:45.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:45.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:46.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:46.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:46.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:46.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:47.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:47.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:47.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:47.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:48.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:48.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:48.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:48.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:49.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:49.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:49.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:49.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:50.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:50.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:50.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:50.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:51.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:51.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:51.338 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:51.336+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:52.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:52.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:52.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:52.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:53.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:53.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:53.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:53.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:54.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:54.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:54.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:54.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:55.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:55.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:55.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:55.360+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:56.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:56.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:56.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:56.360+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:57.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:57.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:57.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:57.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:58.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:58.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:58.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:58.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:59.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:00:59.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:00:59.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:00:59.476+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:00.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:00.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:00.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:00.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:01.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:01.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:01.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:01.464+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:02.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:02.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:02.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:02.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:03.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:03.472+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:03.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:03.528+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:04.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:04.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:04.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:04.548+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:05.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:05.496+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:05.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:05.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:06.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:06.480+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:06.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:06.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:07.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:07.468+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:07.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:07.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:08.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:08.456+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:08.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:08.532+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:09.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:09.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:09.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:09.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:10.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:10.384+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:10.524 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:10.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:11.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:11.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:11.489 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:11.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:12.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:12.404+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:12.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:12.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:13.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:13.444+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:13.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:13.484+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:14.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:14.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:14.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:14.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:15.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:15.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:15.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:15.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:16.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:16.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:16.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:16.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:17.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:17.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:17.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:17.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:18.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:18.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:18.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:18.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:19.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:19.379+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:19.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:19.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:20.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:20.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:20.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:20.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:21.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:21.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:21.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:21.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:22.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:22.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:22.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:22.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:23.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:23.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:23.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:23.443+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:24.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:24.375+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:24.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:24.475+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:25.381 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:25.379+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:25.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:25.467+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:26.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:26.431+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:26.516 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:26.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:27.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:27.431+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:27.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:27.527+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:28.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:28.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:28.521 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:28.519+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:29.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:29.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:29.559 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:29.559+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:30.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:30.415+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:30.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:30.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:31.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:31.431+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:31.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:31.567+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:32.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:32.407+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:32.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:32.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:33.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:33.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:33.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:33.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:34.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:34.375+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:34.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:34.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:35.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:35.331+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:35.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:35.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:36.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:36.315+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:36.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:36.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:37.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:37.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:37.649 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:37.647+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:38.321 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:38.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:38.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:38.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:39.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:39.303+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:39.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:39.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:40.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:40.315+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:40.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:40.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:41.338 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:41.335+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:41.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:41.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:42.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:42.291+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:42.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:42.719+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:43.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:43.303+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:43.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:43.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:44.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:44.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:44.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:44.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:45.283 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:45.283+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:45.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:45.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:46.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:46.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:46.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:46.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:47.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:47.283+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:47.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:47.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:48.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:48.235+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:48.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:48.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:49.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:49.223+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:49.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:49.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:50.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:50.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:50.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:50.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:51.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:51.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:51.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:51.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:52.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:52.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:52.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:52.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:53.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:53.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:53.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:53.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:54.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:54.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:54.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:54.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:55.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:55.211+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:55.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:55.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:56.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:56.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:56.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:56.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:57.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:57.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:57.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:57.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:58.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:58.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:58.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:58.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:59.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:01:59.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:01:59.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:01:59.791+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:00.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:00.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:00.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:00.767+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:01.094 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:01.091+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:01.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:01.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:02.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:02.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:02.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:02.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:03.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:03.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:03.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:03.783+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:04.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:04.087+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:04.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:04.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:05.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:05.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:05.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:05.747+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:06.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:06.035+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:06.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:06.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:07.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:07.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:07.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:07.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:07.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:07.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:08.702 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:08.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:08.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:08.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:09.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:09.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:09.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:09.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:10.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:10.719+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:10.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:10.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:11.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:11.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:11.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:11.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:12.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:12.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:12.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:12.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:13.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:13.774+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:13.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:13.918+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:14.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:14.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:14.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:14.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:15.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:15.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:15.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:15.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:16.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:16.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:16.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:16.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:17.802 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:17.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:17.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:17.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:18.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:18.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:18.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:18.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:19.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:19.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:19.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:19.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:20.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:20.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:20.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:20.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:21.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:21.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:21.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:21.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:22.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:22.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:22.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:22.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:23.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:23.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:23.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:23.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:24.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:24.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:24.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:24.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:25.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:25.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:25.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:25.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:26.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:26.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:26.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:26.930+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:27.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:27.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:27.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:27.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:28.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:28.802+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:28.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:28.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:29.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:29.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:29.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:29.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:30.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:30.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:30.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:30.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:31.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:31.782+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:31.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:31.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:32.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:32.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:32.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:32.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:33.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:33.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:33.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:33.814+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:34.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:34.730+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:34.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:34.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:35.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:35.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:35.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:35.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:36.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:36.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:36.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:36.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:37.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:37.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:37.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:37.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:38.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:38.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:38.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:38.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:39.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:39.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:39.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:39.754+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:40.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:40.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:40.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:40.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:41.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:41.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:41.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:41.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:42.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:42.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:42.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:42.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:43.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:43.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:43.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:43.806+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:44.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:44.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:44.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:44.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:45.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:45.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:45.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:45.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:46.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:46.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:46.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:46.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:47.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:47.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:47.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:47.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:48.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:48.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:48.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:48.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:49.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:49.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:49.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:49.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:50.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:50.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:50.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:50.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:51.847 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:51.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:51.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:51.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:52.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:52.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:52.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:52.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:53.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:53.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:53.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:53.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:54.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:54.806+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:54.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:54.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:55.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:55.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:55.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:55.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:56.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:56.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:56.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:56.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:57.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:57.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:57.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:57.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:58.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:58.802+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:58.937 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:58.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:59.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:02:59.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:02:59.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:02:59.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:00.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:00.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:00.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:00.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:01.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:01.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:01.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:01.878+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:02.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:02.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:02.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:02.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:03.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:03.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:03.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:03.973+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:04.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:04.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:04.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:04.937+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:05.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:05.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:05.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:05.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:06.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:06.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:07.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:07.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:07.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:07.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:08.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:08.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:08.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:08.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:09.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:09.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:09.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:09.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:10.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:10.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:10.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:10.925+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:11.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:11.065+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:11.941 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:11.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:12.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:12.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:12.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:12.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:13.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:13.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:13.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:13.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:14.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:14.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:14.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:14.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:15.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:15.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:15.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:15.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:16.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:16.169+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:16.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:16.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:17.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:17.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:17.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:17.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:18.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:18.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:18.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:18.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:19.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:19.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:19.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:19.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:20.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:20.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:20.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:20.957+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:21.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:21.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:21.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:21.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:22.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:22.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:22.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:22.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:23.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:23.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:23.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:23.993+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:24.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:24.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:24.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:24.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:25.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:25.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:25.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:25.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:26.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:26.185+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:26.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:26.957+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:27.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:27.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:27.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:27.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:28.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:28.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:28.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:28.961+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:29.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:29.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:29.997 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:29.993+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:30.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:30.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:31.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:31.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:31.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:31.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:32.040 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:32.037+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:32.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:32.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:33.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:33.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:33.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:33.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:34.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:34.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:34.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:34.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:35.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:35.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:35.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:35.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:36.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:36.029+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:36.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:36.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:37.023 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:37.021+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:37.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:37.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:38.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:38.001+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:38.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:38.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:38.964 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:38.961+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:39.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:39.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:39.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:39.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:40.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:40.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:40.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:40.957+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:41.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:41.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:41.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:41.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:42.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:42.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:42.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:42.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:43.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:43.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:43.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:43.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:44.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:44.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:44.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:44.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:45.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:45.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:46.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:46.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:46.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:46.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:47.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:47.013+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:47.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:47.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:48.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:48.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:48.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:48.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:49.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:49.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:49.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:49.341+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:50.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:50.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:50.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:50.365+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:51.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:51.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:51.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:51.393+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:52.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:52.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:52.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:52.353+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:52.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:52.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:53.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:53.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:54.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:54.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:54.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:54.325+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:55.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:54.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:55.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:55.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:56.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:55.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:56.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:56.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:56.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:56.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:57.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:57.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:57.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:57.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:58.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:58.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:58.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:58.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:59.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:03:59.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:03:59.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:03:59.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:00.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:00.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:00.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:00.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:01.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:01.224+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:01.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:01.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:02.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:02.216+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:02.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:02.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:03.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:03.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:03.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:03.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:04.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:04.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:04.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:04.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:05.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:05.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:05.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:05.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:06.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:06.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:06.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:06.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:07.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:07.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:07.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:07.964+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:08.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:08.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:08.943 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:08.940+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:09.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:09.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:09.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:09.964+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:10.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:10.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:10.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:10.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:11.326 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:11.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:11.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:11.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:12.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:12.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:12.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:12.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:13.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:13.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:13.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:13.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:14.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:14.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:14.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:14.888+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:15.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:15.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:15.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:15.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:16.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:16.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:16.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:16.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:17.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:17.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:17.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:17.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:18.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:18.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:18.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:18.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:19.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:19.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:19.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:19.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:20.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:20.376+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:20.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:20.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:21.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:21.376+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:21.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:21.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:22.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:22.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:22.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:22.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:23.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:23.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:23.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:23.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:24.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:24.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:24.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:24.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:25.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:25.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:25.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:25.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:26.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:26.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:26.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:26.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:27.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:27.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:27.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:27.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:28.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:28.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:28.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:28.676+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:29.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:29.592+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:29.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:29.656+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:30.613 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:30.608+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:30.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:30.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:31.640 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:31.636+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:31.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:31.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:32.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:32.648+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:32.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:32.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:33.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:33.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:33.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:33.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:34.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:34.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:34.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:34.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:35.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:35.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:35.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:35.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:36.712 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:36.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:36.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:36.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:37.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:37.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:37.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:37.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:38.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:38.744+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:38.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:38.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:39.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:39.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:39.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:39.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:40.815 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:40.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:40.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:40.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:41.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:41.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:41.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:41.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:42.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:42.828+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:42.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:42.904+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:43.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:43.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:43.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:43.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:44.802 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:44.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:44.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:44.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:45.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:45.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:45.930 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:45.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:46.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:46.808+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:46.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:46.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:47.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:47.768+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:47.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:47.900+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:48.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:48.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:48.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:48.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:49.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:49.772+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:49.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:49.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:50.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:50.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:50.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:50.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:51.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:51.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:51.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:51.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:52.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:52.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:52.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:52.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:53.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:53.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:53.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:53.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:54.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:54.759+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:54.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:54.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:55.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:55.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:55.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:55.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:56.752 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:56.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:56.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:56.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:57.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:57.787+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:57.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:57.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:58.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:58.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:58.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:58.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:59.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:04:59.767+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:04:59.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:04:59.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:00.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:00.815+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:00.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:00.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:01.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:01.863+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:01.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:01.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:02.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:02.875+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:02.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:02.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:03.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:03.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:03.873 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:03.871+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:04.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:04.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:04.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:04.919+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:05.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:05.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:05.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:05.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:06.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:06.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:06.951 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:06.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:07.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:07.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:07.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:07.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:08.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:08.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:08.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:08.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:09.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:09.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:09.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:09.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:10.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:10.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:10.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:10.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:11.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:11.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:11.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:11.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:12.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:12.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:12.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:12.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:13.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:13.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:13.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:13.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:14.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:14.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:14.960 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:14.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:15.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:15.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:15.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:15.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:16.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:16.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:16.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:16.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:17.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:17.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:17.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:17.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:18.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:18.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:18.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:18.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:19.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:19.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:19.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:19.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:20.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:20.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:21.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:21.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:21.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:21.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:22.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:22.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:22.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:22.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:23.018 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:23.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:23.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:23.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:24.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:24.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:24.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:24.963+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:24.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:24.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:25.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:25.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:25.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:25.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:26.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:26.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:26.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:26.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:27.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:27.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:27.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:27.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:28.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:28.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:28.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:28.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:29.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:29.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:30.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:30.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:30.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:30.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:31.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:31.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:31.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:31.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:31.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:31.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:32.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:32.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:33.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:33.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:33.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:33.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:34.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:34.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:34.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:34.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:35.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:35.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:35.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:35.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:36.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:36.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:36.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:36.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:37.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:37.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:38.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:38.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:38.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:38.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:39.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:39.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:39.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:39.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:39.990 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:39.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:40.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:40.047+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:41.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:41.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:41.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:41.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:42.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:42.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:42.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:42.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:42.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:42.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:43.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:43.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:44.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:44.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:44.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:44.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:45.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:45.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:45.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:45.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:46.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:46.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:46.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:46.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:47.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:47.022+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:47.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:47.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:48.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:47.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:48.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:48.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:48.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:48.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:49.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:49.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:50.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:50.018+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:50.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:50.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:51.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:51.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:51.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:51.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:52.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:52.006+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:52.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:52.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:53.007 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:53.006+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:53.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:53.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:54.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:54.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:54.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:54.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:54.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:54.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:55.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:55.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:56.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:56.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:56.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:56.174+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:57.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:57.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:57.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:57.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:57.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:57.982+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:58.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:58.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:58.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:05:58.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:05:59.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:05:59.178+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:00.019 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:00.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:00.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:00.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:01.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:01.010+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:01.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:01.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:01.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:01.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:02.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:02.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:03.034 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:03.030+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:03.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:03.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:04.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:04.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:04.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:04.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:05.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:05.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:05.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:05.174+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:06.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:06.042+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:06.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:06.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:07.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:07.074+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:07.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:07.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:08.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:08.026+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:08.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:08.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:08.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:08.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:09.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:09.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:09.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:09.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:10.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:10.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:10.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:10.950+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:11.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:11.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:11.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:11.942+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:12.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:12.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:12.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:12.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:13.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:13.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:13.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:13.934+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:14.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:14.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:14.906 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:14.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:15.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:15.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:15.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:15.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:16.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:16.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:16.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:16.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:17.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:17.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:17.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:17.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:18.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:18.110+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:18.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:18.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:19.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:19.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:19.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:19.774+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:20.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:20.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:20.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:20.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:21.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:21.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:21.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:21.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:22.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:22.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:22.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:22.830+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:23.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:23.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:23.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:23.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:24.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:24.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:24.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:24.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:25.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:25.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:25.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:25.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:26.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:26.230+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:26.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:26.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:27.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:27.190+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:27.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:27.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:28.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:28.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:28.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:28.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:29.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:29.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:29.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:29.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:30.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:30.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:30.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:30.802+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:31.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:31.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:31.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:31.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:32.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:32.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:32.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:32.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:33.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:33.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:33.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:33.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:34.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:34.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:34.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:34.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:35.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:35.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:35.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:35.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:36.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:36.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:36.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:36.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:37.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:37.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:37.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:37.909+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:38.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:38.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:38.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:38.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:39.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:39.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:39.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:39.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:40.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:40.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:40.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:40.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:41.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:41.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:41.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:41.933+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:42.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:42.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:42.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:42.957+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:43.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:43.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:43.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:43.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:44.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:44.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:44.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:44.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:45.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:45.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:45.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:45.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:46.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:46.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:47.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:47.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:47.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:47.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:47.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:47.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:48.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:48.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:48.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:48.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:49.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:49.353+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:49.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:49.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:50.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:50.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:50.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:50.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:51.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:51.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:51.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:51.905+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:52.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:52.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:52.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:52.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:53.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:53.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:53.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:53.905+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:54.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:54.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:54.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:54.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:55.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:55.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:55.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:55.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:56.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:56.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:56.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:56.853+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:57.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:57.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:57.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:57.889+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:58.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:58.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:58.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:58.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:59.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:06:59.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:06:59.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:06:59.925+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:00.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:00.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:00.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:00.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:01.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:01.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:01.978 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:01.973+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:02.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:02.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:02.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:02.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:03.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:03.341+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:03.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:03.973+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:04.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:04.325+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:04.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:04.957+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:05.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:05.369+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:05.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:05.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:06.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:06.369+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:06.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:06.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:07.337 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:07.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:08.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:08.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:08.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:08.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:08.971 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:08.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:09.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:09.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:09.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:09.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:10.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:10.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:11.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:11.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:11.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:11.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:12.017 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:12.013+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:12.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:12.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:12.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:12.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:13.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:13.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:13.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:13.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:14.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:14.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:14.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:14.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:15.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:15.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:15.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:15.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:16.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:16.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:16.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:16.957+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:17.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:17.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:18.005 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:18.001+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:18.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:18.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:18.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:18.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:19.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:19.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:20.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:20.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:20.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:20.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:21.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:21.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:21.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:21.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:22.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:22.005+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:22.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:22.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:23.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:23.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:23.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:23.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:24.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:24.045+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:24.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:24.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:25.014 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:25.009+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:25.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:25.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:26.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:26.049+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:26.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:26.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:27.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:27.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:27.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:27.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:28.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:28.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:28.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:28.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:29.128 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:29.124+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:29.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:29.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:30.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:30.084+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:30.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:30.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:31.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:31.052+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:31.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:31.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:32.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:32.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:32.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:32.248+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:33.077 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:33.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:33.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:33.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:34.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:34.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:34.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:34.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:35.054 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:35.052+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:35.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:35.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:36.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:36.028+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:36.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:36.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:37.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:37.064+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:37.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:37.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:38.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:38.020+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:38.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:38.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:39.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:39.052+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:39.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:39.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:40.029 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:40.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:40.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:40.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:41.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:41.020+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:41.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:41.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:41.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:41.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:42.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:42.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:42.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:42.988+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:43.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:43.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:43.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:43.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:44.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:44.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:45.015 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:45.012+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:45.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:45.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:46.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:46.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:46.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:46.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:47.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:47.020+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:47.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:47.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:48.060 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:48.056+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:48.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:48.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:49.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:49.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:49.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:49.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:50.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:50.096+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:50.337 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:50.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:51.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:51.056+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:51.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:51.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:52.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:52.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:52.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:52.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:53.093 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:53.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:53.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:53.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:54.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:54.076+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:54.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:54.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:55.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:55.076+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:55.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:55.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:56.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:56.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:56.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:56.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:57.103 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:57.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:57.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:57.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:58.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:58.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:58.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:58.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:59.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:07:59.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:07:59.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:07:59.408+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:00.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:00.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:00.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:00.408+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:01.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:01.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:01.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:01.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:02.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:02.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:02.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:02.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:03.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:03.104+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:03.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:03.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:04.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:04.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:04.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:04.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:05.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:05.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:05.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:05.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:06.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:06.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:06.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:06.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:07.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:07.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:07.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:07.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:08.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:08.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:08.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:08.528+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:09.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:09.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:09.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:09.528+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:10.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:10.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:10.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:10.556+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:11.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:11.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:11.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:11.512+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:12.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:12.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:12.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:12.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:13.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:13.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:13.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:13.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:14.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:14.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:14.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:14.532+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:15.094 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:15.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:15.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:15.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:16.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:16.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:16.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:16.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:17.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:17.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:17.540 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:17.540+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:18.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:18.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:18.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:18.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:19.110 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:19.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:19.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:19.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:20.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:20.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:20.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:20.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:21.132 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:21.132+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:21.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:21.536+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:22.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:22.119+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:22.545 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:22.543+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:23.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:23.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:23.528 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:23.527+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:24.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:24.135+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:24.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:24.575+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:25.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:25.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:25.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:25.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:26.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:26.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:26.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:26.583+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:27.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:27.167+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:27.580 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:27.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:28.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:28.167+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:28.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:28.567+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:29.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:29.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:29.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:29.575+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:30.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:30.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:30.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:30.531+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:31.191 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:31.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:31.569 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:31.567+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:32.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:32.223+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:32.598 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:32.595+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:33.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:33.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:33.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:33.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:34.258 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:34.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:34.599 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:34.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:35.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:35.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:35.580 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:35.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:36.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:36.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:36.536 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:36.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:37.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:37.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:37.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:37.559+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:38.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:38.223+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:38.576 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:38.575+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:39.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:39.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:39.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:39.595+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:40.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:40.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:40.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:40.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:41.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:41.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:41.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:41.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:42.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:42.299+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:42.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:42.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:43.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:43.279+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:43.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:43.611+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:44.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:44.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:44.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:44.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:45.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:45.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:45.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:45.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:46.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:46.235+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:46.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:46.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:47.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:47.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:47.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:47.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:48.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:48.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:48.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:48.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:49.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:49.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:49.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:49.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:50.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:50.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:50.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:50.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:51.121 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:51.119+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:51.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:51.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:52.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:52.135+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:52.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:52.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:53.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:53.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:53.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:53.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:54.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:54.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:54.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:54.771+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:55.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:55.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:55.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:55.783+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:56.142 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:56.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:56.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:56.799+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:57.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:57.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:57.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:57.815+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:58.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:58.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:58.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:58.803+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:59.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:08:59.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:08:59.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:08:59.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:00.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:00.123+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:00.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:00.767+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:01.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:01.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:01.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:01.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:02.133 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:02.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:02.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:02.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:03.134 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:03.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:03.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:03.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:04.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:04.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:04.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:04.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:05.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:05.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:05.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:05.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:06.101 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:06.099+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:06.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:06.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:07.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:07.127+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:07.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:07.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:08.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:08.095+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:08.675 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:08.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:09.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:09.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:09.663 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:09.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:10.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:10.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:10.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:10.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:11.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:11.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:11.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:11.667+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:12.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:12.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:12.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:12.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:13.041 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:13.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:13.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:13.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:14.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:14.083+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:14.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:14.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:15.126 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:15.122+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:15.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:15.658+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:16.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:16.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:16.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:16.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:17.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:17.166+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:17.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:17.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:18.132 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:18.130+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:18.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:18.750+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:19.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:19.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:19.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:19.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:20.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:20.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:20.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:20.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:21.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:21.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:21.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:21.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:22.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:22.118+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:22.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:22.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:23.096 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:23.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:23.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:23.814+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:24.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:24.062+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:24.820 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:24.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:25.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:25.110+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:25.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:25.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:26.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:26.118+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:26.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:26.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:27.133 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:27.130+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:27.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:27.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:28.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:28.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:28.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:28.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:29.169 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:29.166+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:29.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:29.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:30.137 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:30.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:30.835 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:30.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:31.094 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:31.090+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:31.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:31.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:32.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:32.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:32.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:32.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:33.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:33.154+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:33.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:33.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:34.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:34.154+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:34.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:34.890+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:35.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:35.122+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:35.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:35.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:36.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:36.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:36.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:36.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:37.137 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:37.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:37.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:37.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:38.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:38.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:38.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:38.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:39.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:39.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:39.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:39.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:40.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:40.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:41.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:41.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:41.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:41.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:42.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:42.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:42.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:42.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:43.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:43.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:43.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:43.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:44.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:44.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:44.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:44.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:45.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:45.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:45.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:45.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:46.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:46.154+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:46.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:46.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:47.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:47.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:47.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:47.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:48.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:48.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:48.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:48.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:49.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:49.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:49.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:49.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:50.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:50.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:50.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:50.290+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:51.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:51.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:51.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:51.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:52.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:52.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:52.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:52.314+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:53.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:53.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:53.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:53.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:54.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:54.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:54.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:54.346+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:55.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:55.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:55.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:55.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:56.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:56.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:56.385 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:56.382+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:57.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:57.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:57.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:57.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:58.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:58.058+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:58.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:58.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:59.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:09:59.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:09:59.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:09:59.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:00.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:00.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:00.373 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:00.370+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:01.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:01.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:01.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:01.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:02.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:02.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:02.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:02.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:03.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:03.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:03.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:03.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:04.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:04.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:04.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:04.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:05.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:05.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:05.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:05.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:06.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:06.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:06.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:06.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:07.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:07.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:07.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:07.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:08.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:08.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:08.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:08.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:09.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:09.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:09.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:09.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:10.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:10.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:10.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:10.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:11.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:11.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:11.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:11.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:12.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:12.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:12.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:12.309+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:13.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:13.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:13.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:13.277+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:14.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:14.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:14.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:14.269+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:15.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:15.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:15.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:15.249+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:16.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:16.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:16.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:16.237+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:17.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:17.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:17.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:17.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:18.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:18.241+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:18.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:18.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:19.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:19.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:19.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:19.261+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:20.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:20.221+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:20.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:20.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:21.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:21.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:21.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:21.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:22.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:22.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:22.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:22.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:23.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:23.141+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:23.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:23.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:24.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:24.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:24.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:24.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:25.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:25.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:25.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:25.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:26.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:26.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:26.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:26.209+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:27.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:27.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:27.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:27.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:28.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:28.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:28.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:28.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:29.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:29.093+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:29.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:29.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:30.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:30.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:30.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:30.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:31.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:31.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:31.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:31.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:32.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:32.081+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:32.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:32.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:33.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:33.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:33.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:33.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:34.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:34.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:34.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:34.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:35.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:35.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:35.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:35.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:36.067 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:36.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:36.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:36.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:37.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:37.069+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:37.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:37.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:38.057 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:38.053+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:38.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:38.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:39.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:39.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:39.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:39.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:40.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:40.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:40.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:40.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:41.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:41.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:41.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:41.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:42.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:42.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:42.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:42.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:43.080 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:43.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:43.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:43.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:44.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:44.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:44.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:44.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:45.059 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:45.057+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:45.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:45.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:46.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:46.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:46.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:46.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:47.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:47.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:47.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:47.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:48.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:48.097+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:48.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:48.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:49.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:49.093+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:49.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:49.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:50.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:50.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:50.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:50.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:51.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:51.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:51.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:51.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:52.125 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:52.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:52.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:52.349+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:53.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:53.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:53.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:53.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:54.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:54.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:54.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:54.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:55.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:55.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:55.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:55.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:56.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:56.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:56.366 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:56.361+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:57.133 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:57.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:57.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:57.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:58.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:58.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:58.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:58.349+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:59.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:10:59.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:10:59.335 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:10:59.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:00.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:00.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:00.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:00.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:01.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:01.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:01.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:01.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:02.251 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:02.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:02.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:02.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:03.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:03.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:03.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:03.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:04.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:04.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:04.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:04.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:05.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:05.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:05.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:05.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:06.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:06.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:06.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:06.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:07.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:07.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:07.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:07.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:08.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:08.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:08.539 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:08.536+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:09.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:09.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:09.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:09.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:10.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:10.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:10.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:10.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:11.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:11.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:11.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:11.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:12.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:12.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:12.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:12.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:13.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:13.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:13.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:13.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:14.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:14.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:14.402 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:14.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:15.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:15.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:15.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:15.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:16.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:16.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:16.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:16.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:17.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:17.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:17.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:17.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:18.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:18.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:18.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:18.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:19.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:19.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:19.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:19.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:20.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:20.356+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:20.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:20.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:21.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:21.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:21.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:21.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:22.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:22.348+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:22.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:22.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:23.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:23.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:23.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:23.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:24.321 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:24.316+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:24.434 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:24.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:25.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:25.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:25.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:25.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:26.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:26.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:26.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:26.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:27.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:27.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:27.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:27.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:28.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:28.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:28.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:28.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:29.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:29.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:29.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:29.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:30.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:30.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:30.380 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:30.376+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:31.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:31.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:31.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:31.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:32.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:32.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:32.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:32.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:33.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:33.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:33.337 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:33.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:34.224 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:34.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:34.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:34.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:35.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:35.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:35.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:35.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:36.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:36.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:36.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:36.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:37.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:37.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:37.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:37.248+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:38.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:38.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:38.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:38.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:39.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:39.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:39.258 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:39.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:40.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:40.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:40.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:40.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:41.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:41.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:41.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:41.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:42.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:42.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:42.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:42.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:43.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:43.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:43.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:43.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:44.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:44.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:44.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:44.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:45.209 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:45.208+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:45.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:45.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:46.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:46.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:46.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:46.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:47.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:47.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:47.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:47.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:48.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:48.164+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:48.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:48.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:49.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:49.160+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:49.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:49.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:50.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:50.204+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:50.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:50.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:51.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:51.172+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:51.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:51.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:52.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:52.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:52.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:52.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:53.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:53.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:53.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:53.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:54.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:54.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:54.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:54.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:55.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:55.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:55.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:55.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:56.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:56.167+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:56.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:56.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:57.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:57.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:57.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:57.315+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:58.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:58.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:58.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:58.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:59.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:11:59.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:11:59.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:11:59.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:00.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:00.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:00.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:00.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:01.123 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:01.123+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:01.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:01.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:02.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:02.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:02.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:02.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:03.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:03.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:03.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:03.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:04.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:04.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:04.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:04.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:05.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:05.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:05.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:05.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:06.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:06.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:06.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:06.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:07.110 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:07.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:07.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:07.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:08.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:08.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:08.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:08.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:09.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:09.119+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:09.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:09.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:10.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:10.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:10.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:10.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:11.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:11.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:11.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:11.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:12.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:12.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:12.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:12.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:13.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:13.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:13.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:13.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:14.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:14.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:14.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:14.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:15.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:15.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:15.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:15.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:16.120 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:16.119+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:16.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:16.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:17.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:17.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:17.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:17.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:18.134 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:18.135+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:18.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:18.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:19.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:19.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:19.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:19.315+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:20.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:20.107+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:20.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:20.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:21.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:21.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:21.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:21.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:22.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:22.159+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:22.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:22.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:23.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:23.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:23.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:23.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:24.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:24.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:24.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:24.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:25.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:25.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:25.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:25.415+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:26.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:26.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:26.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:26.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:27.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:27.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:27.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:27.499+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:28.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:28.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:28.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:28.531+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:29.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:29.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:29.540 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:29.539+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:30.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:30.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:30.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:30.575+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:31.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:31.291+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:31.589 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:31.587+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:32.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:32.311+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:32.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:32.595+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:33.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:33.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:33.623 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:33.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:34.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:34.303+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:34.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:34.587+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:35.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:35.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:35.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:35.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:36.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:36.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:36.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:36.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:37.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:37.307+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:37.660 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:37.659+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:38.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:38.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:38.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:38.659+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:39.310 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:39.307+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:39.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:39.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:40.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:40.307+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:40.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:40.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:41.358 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:41.355+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:41.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:41.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:42.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:42.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:42.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:42.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:43.318 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:43.315+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:43.720 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:43.719+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:44.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:44.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:44.735 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:44.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:45.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:45.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:45.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:45.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:46.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:46.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:46.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:46.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:47.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:47.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:47.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:47.774+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:48.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:48.286+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:48.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:48.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:49.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:49.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:49.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:49.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:50.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:50.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:50.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:50.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:51.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:51.222+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:51.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:51.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:52.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:52.174+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:52.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:52.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:53.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:53.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:53.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:53.642+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:54.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:54.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:54.695 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:54.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:55.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:55.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:55.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:55.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:56.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:56.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:56.647 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:56.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:57.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:57.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:57.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:57.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:58.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:58.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:58.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:58.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:59.283 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:12:59.282+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:12:59.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:12:59.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:00.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:00.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:00.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:00.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:01.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:01.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:01.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:01.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:02.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:02.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:02.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:02.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:03.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:03.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:03.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:03.726+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:04.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:04.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:04.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:04.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:05.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:05.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:05.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:05.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:06.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:06.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:06.826 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:06.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:07.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:07.302+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:07.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:07.774+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:08.261 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:08.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:08.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:08.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:09.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:09.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:09.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:09.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:10.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:10.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:10.835 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:10.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:11.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:11.262+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:11.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:11.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:12.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:12.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:12.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:12.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:13.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:13.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:13.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:13.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:14.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:14.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:14.825 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:14.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:15.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:15.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:15.825 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:15.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:16.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:16.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:16.874 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:16.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:17.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:17.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:17.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:17.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:18.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:18.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:18.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:18.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:19.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:19.154+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:19.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:19.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:20.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:20.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:20.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:20.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:21.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:21.178+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:21.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:21.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:22.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:22.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:22.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:22.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:23.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:23.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:23.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:23.886+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:24.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:24.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:24.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:24.878+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:25.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:25.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:25.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:25.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:26.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:26.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:26.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:26.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:27.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:27.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:27.810 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:27.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:28.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:28.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:28.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:28.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:29.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:29.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:29.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:29.766+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:30.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:30.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:30.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:30.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:31.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:31.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:31.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:31.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:32.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:32.282+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:32.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:32.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:33.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:33.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:33.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:33.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:34.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:34.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:34.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:34.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:35.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:35.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:35.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:35.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:36.321 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:36.318+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:36.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:36.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:37.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:37.286+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:37.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:37.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:38.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:38.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:38.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:38.709+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:39.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:39.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:39.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:39.681+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:40.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:40.289+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:40.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:40.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:41.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:41.245+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:41.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:41.605+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:42.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:42.253+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:42.574 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:42.573+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:43.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:43.249+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:43.534 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:43.533+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:44.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:44.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:44.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:44.485+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:45.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:45.237+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:45.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:45.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:46.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:46.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:46.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:46.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:47.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:47.233+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:47.459 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:47.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:48.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:48.197+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:48.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:48.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:49.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:49.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:49.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:49.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:50.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:50.197+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:50.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:50.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:51.174 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:51.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:51.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:51.513+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:52.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:52.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:52.558 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:52.553+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:53.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:53.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:53.541 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:53.537+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:54.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:54.257+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:54.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:54.589+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:55.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:55.249+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:55.622 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:55.617+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:56.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:56.297+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:56.617 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:56.613+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:57.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:57.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:57.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:57.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:58.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:58.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:58.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:58.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:59.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:13:59.313+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:13:59.678 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:13:59.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:00.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:00.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:00.678 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:00.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:01.369 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:01.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:01.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:01.685+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:02.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:02.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:02.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:02.693+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:03.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:03.313+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:03.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:03.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:04.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:04.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:04.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:04.729+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:05.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:05.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:05.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:05.729+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:06.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:06.421+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:06.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:06.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:07.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:07.421+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:07.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:07.753+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:08.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:08.421+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:08.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:08.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:09.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:09.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:09.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:09.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:10.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:10.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:10.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:10.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:11.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:11.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:11.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:11.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:12.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:12.417+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:12.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:12.849+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:13.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:13.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:13.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:13.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:14.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:14.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:14.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:14.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:15.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:15.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:15.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:15.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:16.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:16.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:16.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:16.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:17.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:17.497+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:17.862 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:17.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:18.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:18.541+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:18.908 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:18.905+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:19.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:19.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:19.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:19.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:20.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:20.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:20.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:20.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:21.615 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:21.613+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:21.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:21.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:22.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:22.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:22.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:22.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:23.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:23.593+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:23.862 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:23.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:24.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:24.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:24.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:24.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:25.649 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:25.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:25.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:25.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:26.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:26.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:26.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:26.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:27.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:27.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:27.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:27.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:28.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:28.673+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:28.866 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:28.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:29.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:29.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:29.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:29.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:30.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:30.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:30.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:30.837+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:31.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:31.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:31.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:31.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:32.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:32.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:32.914 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:32.912+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:33.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:33.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:33.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:33.908+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:34.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:34.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:34.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:34.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:35.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:35.836+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:36.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:35.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:36.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:36.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:37.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:37.000+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:37.907 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:37.904+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:37.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:37.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:38.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:38.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:39.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:39.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:39.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:39.900+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:40.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:39.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:40.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:40.944+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:41.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:41.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:41.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:41.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:42.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:42.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:42.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:42.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:42.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:42.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:43.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:43.964+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:43.978 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:43.972+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:44.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:44.952+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:44.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:44.964+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:45.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:45.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:45.934 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:45.928+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:46.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:46.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:46.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:46.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:47.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:47.928+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:47.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:47.952+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:48.893 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:48.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:48.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:48.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:49.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:49.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:49.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:49.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:50.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:50.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:50.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:50.920+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:51.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:51.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:51.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:51.940+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:52.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:52.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:52.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:52.936+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:53.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:53.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:53.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:53.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:54.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:54.940+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:54.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:54.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:55.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:55.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:55.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:55.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:56.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:56.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:56.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:56.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:57.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:57.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:57.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:57.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:58.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:58.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:58.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:58.912+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:59.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:14:59.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:14:59.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:14:59.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:00.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:00.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:00.963 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:00.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:01.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:01.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:01.917 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:01.912+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:02.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:02.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:02.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:02.904+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:03.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:03.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:03.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:03.940+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:04.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:04.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:04.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:04.964+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:05.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:05.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:05.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:05.920+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:06.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:06.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:06.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:06.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:07.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:07.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:07.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:07.876+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:08.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:08.872+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:08.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:08.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:09.847 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:09.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:09.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:09.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:10.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:10.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:10.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:10.892+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:11.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:11.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:11.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:11.928+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:12.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:12.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:12.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:12.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:13.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:13.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:13.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:13.884+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:14.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:14.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:14.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:14.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:15.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:15.836+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:15.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:15.912+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:16.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:16.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:16.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:16.908+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:17.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:17.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:17.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:17.948+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:18.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:18.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:18.933 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:18.932+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:19.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:19.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:19.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:19.984+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:20.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:20.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:20.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:20.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:21.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:21.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:21.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:21.956+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:22.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:22.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:22.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:22.956+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:23.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:23.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:24.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:23.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:24.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:24.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:25.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:25.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:25.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:25.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:25.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:25.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:26.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:26.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:27.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:27.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:27.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:27.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:27.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:27.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:28.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:28.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:29.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:29.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:29.936 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:29.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:30.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:30.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:30.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:30.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:30.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:30.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:31.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:31.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:31.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:31.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:32.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:32.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:32.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:32.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:33.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:33.915+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:33.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:33.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:34.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:34.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:34.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:34.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:35.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:35.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:35.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:35.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:36.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:36.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:36.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:36.951+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:37.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:37.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:37.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:37.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:38.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:38.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:38.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:38.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:39.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:39.783+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:39.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:39.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:40.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:40.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:40.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:40.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:41.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:41.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:42.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:42.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:42.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:42.719+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:42.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:42.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:43.719 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:43.719+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:43.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:43.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:44.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:44.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:44.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:44.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:45.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:45.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:45.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:45.975+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:46.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:46.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:46.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:46.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:47.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:47.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:47.949 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:47.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:48.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:48.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:48.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:48.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:49.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:49.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:49.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:49.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:50.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:50.723+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:50.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:50.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:51.718 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:51.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:51.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:51.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:52.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:52.683+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:52.917 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:52.915+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:53.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:53.699+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:53.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:53.895+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:54.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:54.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:54.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:54.863+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:55.668 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:55.667+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:55.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:55.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:56.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:56.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:56.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:56.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:57.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:57.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:57.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:57.787+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:58.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:58.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:58.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:58.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:59.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:15:59.699+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:15:59.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:15:59.719+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:00.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:00.667+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:00.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:00.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:01.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:01.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:01.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:01.747+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:02.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:02.663+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:02.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:02.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:03.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:03.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:03.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:03.759+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:04.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:04.719+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:04.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:04.791+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:05.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:05.699+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:05.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:05.747+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:06.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:06.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:06.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:06.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:07.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:07.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:07.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:07.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:08.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:08.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:08.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:08.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:09.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:09.647+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:09.682 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:09.679+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:10.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:10.643+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:10.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:10.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:11.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:11.667+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:11.682 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:11.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:12.635 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:12.635+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:12.636 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:12.635+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:13.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:13.611+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:13.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:13.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:14.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:14.587+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:14.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:14.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:15.578 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:15.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:15.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:15.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:16.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:16.614+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:16.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:16.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:17.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:17.614+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:17.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:17.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:18.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:18.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:18.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:18.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:19.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:19.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:19.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:19.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:20.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:20.562+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:20.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:20.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:21.554 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:21.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:21.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:21.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:22.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:22.602+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:22.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:22.682+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:23.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:23.650+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:23.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:23.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:24.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:24.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:24.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:24.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:25.640 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:25.638+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:25.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:25.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:26.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:26.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:26.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:26.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:27.598 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:27.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:27.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:27.794+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:28.633 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:28.630+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:28.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:28.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:29.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:29.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:29.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:29.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:30.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:30.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:30.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:30.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:31.603 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:31.602+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:31.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:31.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:32.559 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:32.558+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:32.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:32.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:33.516 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:33.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:33.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:33.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:34.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:34.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:34.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:34.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:35.499 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:35.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:35.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:35.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:36.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:36.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:36.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:36.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:37.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:37.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:37.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:37.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:38.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:38.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:38.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:38.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:39.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:39.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:39.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:39.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:40.579 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:40.578+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:40.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:40.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:41.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:41.570+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:41.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:41.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:42.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:42.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:42.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:42.602+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:43.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:43.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:43.606 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:43.602+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:44.543 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:44.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:44.610 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:44.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:45.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:45.518+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:45.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:45.598+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:46.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:46.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:46.625 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:46.622+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:47.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:47.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:47.605 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:47.602+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:48.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:48.562+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:48.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:48.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:49.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:49.566+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:49.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:49.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:50.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:50.562+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:50.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:50.642+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:51.594 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:51.590+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:51.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:51.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:52.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:52.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:52.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:52.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:53.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:53.626+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:53.698 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:53.694+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:54.636 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:54.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:54.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:54.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:55.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:55.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:55.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:55.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:56.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:56.646+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:56.705 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:56.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:57.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:57.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:57.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:57.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:58.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:58.682+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:58.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:58.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:59.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:16:59.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:16:59.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:16:59.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:00.739 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:00.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:00.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:00.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:01.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:01.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:01.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:01.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:02.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:02.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:02.803 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:02.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:03.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:03.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:03.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:03.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:04.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:04.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:04.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:04.786+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:05.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:05.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:05.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:05.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:06.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:06.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:06.825 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:06.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:07.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:07.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:07.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:07.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:08.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:08.697+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:08.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:08.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:09.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:09.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:09.808 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:09.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:10.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:10.653+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:10.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:10.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:11.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:11.701+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:11.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:11.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:12.668 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:12.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:12.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:12.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:13.626 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:13.625+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:13.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:13.729+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:14.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:14.637+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:14.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:14.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:15.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:15.609+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:15.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:15.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:16.653 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:16.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:16.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:16.821+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:17.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:17.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:17.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:17.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:18.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:18.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:18.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:18.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:19.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:19.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:19.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:19.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:20.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:20.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:20.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:20.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:21.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:21.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:21.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:21.909+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:22.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:22.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:22.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:22.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:23.671 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:23.669+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:23.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:23.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:24.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:24.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:24.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:24.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:25.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:25.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:25.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:25.837+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:26.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:26.697+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:26.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:26.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:27.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:27.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:27.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:27.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:28.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:28.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:28.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:28.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:29.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:29.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:29.887 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:29.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:30.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:30.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:30.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:30.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:31.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:31.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:31.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:31.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:32.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:32.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:32.857 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:32.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:33.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:33.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:33.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:33.881+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:34.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:34.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:34.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:34.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:35.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:35.845+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:35.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:35.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:36.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:36.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:36.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:36.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:37.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:37.889+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:37.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:37.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:38.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:38.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:38.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:38.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:39.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:39.977+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:39.984 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:39.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:40.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:40.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:40.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:40.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:41.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:41.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:42.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:41.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:42.927 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:42.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:43.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:43.001+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:43.893 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:43.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:44.028 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:44.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:44.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:44.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:45.030 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:45.025+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:45.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:45.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:46.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:46.029+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:47.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:47.029+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:47.051 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:47.049+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:48.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:47.997+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:48.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:48.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:49.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:49.017+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:49.126 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:49.121+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:49.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:49.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:50.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:50.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:50.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:50.953+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:51.094 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:51.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:51.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:51.913+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:52.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:52.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:52.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:52.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:53.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:53.097+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:53.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:53.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:54.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:54.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:54.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:54.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:55.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:55.049+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:55.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:55.913+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:56.063 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:56.061+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:56.910 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:56.905+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:57.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:57.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:57.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:57.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:58.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:58.105+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:58.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:58.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:59.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:17:59.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:17:59.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:17:59.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:00.134 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:00.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:00.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:00.981+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:01.132 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:01.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:02.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:02.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:02.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:02.100+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:03.043 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:03.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:03.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:03.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:04.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:04.076+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:04.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:04.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:05.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:05.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:05.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:05.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:06.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:06.088+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:06.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:06.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:07.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:07.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:07.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:07.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:08.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:08.068+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:08.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:08.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:09.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:09.104+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:09.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:09.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:10.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:10.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:10.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:10.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:11.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:11.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:11.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:11.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:12.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:12.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:12.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:12.156+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:13.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:13.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:13.170 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:13.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:14.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:14.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:14.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:14.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:15.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:15.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:15.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:15.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:16.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:16.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:16.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:16.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:17.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:17.092+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:17.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:17.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:18.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:18.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:18.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:18.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:19.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:19.092+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:19.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:19.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:20.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:20.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:20.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:20.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:21.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:21.100+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:21.169 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:21.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:22.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:22.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:22.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:22.140+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:23.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:23.080+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:23.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:23.120+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:24.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:24.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:24.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:24.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:25.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:25.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:25.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:25.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:25.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:25.984+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:26.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:26.148+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:26.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:26.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:27.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:27.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:27.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:27.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:28.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:28.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:28.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:28.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:29.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:29.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:30.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:30.000+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:30.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:30.104+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:31.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:31.012+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:31.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:31.080+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:31.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:31.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:32.132 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:32.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:33.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:33.016+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:33.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:33.116+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:34.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:34.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:34.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:34.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:35.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:35.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:35.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:35.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:35.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:35.972+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:36.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:36.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:36.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:36.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:37.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:37.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:37.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:37.956+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:38.168 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:38.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:38.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:38.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:39.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:39.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:39.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:39.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:40.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:40.192+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:41.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:41.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:41.172 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:41.172+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:42.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:42.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:42.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:42.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:43.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:43.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:43.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:43.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:44.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:44.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:44.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:44.120+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:45.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:45.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:45.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:45.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:46.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:46.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:46.159 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:46.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:47.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:47.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:47.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:47.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:48.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:48.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:48.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:48.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:49.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:49.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:49.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:49.120+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:50.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:50.036+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:50.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:50.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:51.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:51.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:51.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:51.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:51.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:51.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:52.163 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:52.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:52.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:52.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:53.145 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:53.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:54.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:54.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:54.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:54.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:55.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:55.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:55.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:55.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:55.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:55.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:56.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:56.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:57.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:57.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:57.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:57.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:58.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:58.031+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:58.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:58.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:58.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:18:58.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:18:59.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:18:59.251+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:00.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:00.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:00.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:00.211+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:00.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:00.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:01.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:01.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:01.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:01.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:02.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:02.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:03.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:03.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:03.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:03.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:04.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:04.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:04.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:04.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:05.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:05.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:05.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:05.171+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:06.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:06.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:06.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:06.127+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:07.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:07.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:07.132 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:07.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:08.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:08.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:08.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:08.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:09.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:09.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:09.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:09.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:10.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:10.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:10.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:10.167+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:11.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:11.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:11.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:11.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:12.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:12.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:12.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:12.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:13.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:13.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:13.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:13.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:14.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:14.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:14.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:14.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:15.059 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:15.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:15.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:15.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:16.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:16.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:16.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:16.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:17.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:17.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:17.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:17.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:18.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:18.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:18.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:18.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:19.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:19.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:19.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:19.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:20.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:20.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:20.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:20.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:21.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:21.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:21.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:21.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:22.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:22.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:22.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:22.235+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:23.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:23.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:23.223 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:23.223+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:24.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:24.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:24.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:24.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:25.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:25.047+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:25.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:25.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:26.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:26.003+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:26.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:26.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:26.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:26.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:27.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:27.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:27.962 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:27.959+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:28.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:28.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:28.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:28.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:29.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:29.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:30.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:30.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:30.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:30.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:31.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:30.999+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:31.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:31.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:32.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:32.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:32.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:32.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:33.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:33.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:33.114 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:33.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:33.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:33.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:34.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:34.127+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:34.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:34.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:35.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:35.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:35.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:35.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:36.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:36.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:37.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:37.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:37.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:37.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:38.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:38.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:38.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:38.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:39.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:39.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:39.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:39.211+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:40.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:40.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:40.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:40.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:41.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:41.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:41.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:41.219+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:42.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:42.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:42.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:42.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:43.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:43.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:43.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:43.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:44.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:44.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:44.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:44.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:45.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:45.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:45.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:45.231+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:45.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:45.975+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:46.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:46.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:47.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:47.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:47.225 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:47.222+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:48.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:48.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:48.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:48.226+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:48.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:48.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:49.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:49.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:49.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:49.930+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:50.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:50.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:50.905 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:50.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:51.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:51.246+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:51.862 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:51.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:52.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:52.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:52.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:52.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:53.258 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:53.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:53.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:53.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:54.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:54.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:54.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:54.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:55.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:55.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:55.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:55.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:56.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:56.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:56.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:56.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:57.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:57.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:57.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:57.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:58.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:58.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:58.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:19:58.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:19:59.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:19:59.222+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:00.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:00.034+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:00.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:00.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:01.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:01.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:01.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:01.186+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:02.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:02.042+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:02.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:02.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:03.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:03.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:03.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:03.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:04.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:04.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:04.102 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:04.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:05.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:05.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:05.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:05.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:06.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:06.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:06.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:06.154+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:07.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:07.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:07.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:07.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:08.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:08.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:08.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:08.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:09.070 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:09.066+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:09.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:09.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:10.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:10.102+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:10.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:10.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:10.773 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-12T00:20:10.770+0000 7fd5e8fc6700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x7fd5cc05dcf0 msgr2=0x7fd5cc0b0c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-12T00:20:11.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:11.130+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:11.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:11.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:12.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:12.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:12.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:12.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:13.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:13.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:13.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:13.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:14.169 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:14.166+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:14.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:14.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:15.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:15.134+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:15.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:15.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:16.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:16.090+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:16.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:16.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:17.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:17.086+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:17.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:17.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:18.119 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:18.118+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:18.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:18.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:19.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:19.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:19.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:19.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:20.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:20.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:20.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:20.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:21.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:21.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:21.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:21.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:22.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:22.174+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:22.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:22.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:23.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:23.226+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:23.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:23.274+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:24.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:24.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:24.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:24.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:25.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:25.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:25.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:25.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:26.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:26.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:26.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:26.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:27.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:27.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:27.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:27.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:28.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:28.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:28.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:28.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:29.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:29.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:29.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:29.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:30.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:30.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:30.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:30.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:31.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:31.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:31.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:31.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:32.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:32.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:32.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:32.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:33.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:33.182+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:33.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:33.338+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:34.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:34.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:34.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:34.362+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:35.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:35.158+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:35.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:35.386+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:36.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:36.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:36.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:36.414+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:37.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:37.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:37.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:37.450+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:38.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:38.138+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:38.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:38.482+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:39.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:39.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:39.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:39.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:40.133 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:40.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:40.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:40.405+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:41.139 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:41.137+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:41.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:41.413+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:42.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:42.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:42.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:42.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:43.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:43.189+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:43.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:43.505+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:44.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:44.205+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:44.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:44.553+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:45.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:45.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:45.568 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:45.565+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:46.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:46.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:46.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:46.609+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:47.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:47.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:47.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:47.653+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:48.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:48.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:48.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:48.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:49.130 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:49.125+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:49.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:49.625+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:50.094 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:50.089+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:50.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:50.617+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:51.136 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:51.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:51.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:51.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:52.138 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:52.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:52.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:52.665+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:53.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:53.149+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:53.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:53.653+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:54.157 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:54.153+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:54.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:54.605+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:55.203 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:55.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:55.638 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:55.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:56.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:56.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:56.624 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:56.621+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:57.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:57.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:57.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:57.653+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:58.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:58.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:58.702 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:58.697+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:59.135 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:20:59.133+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:20:59.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:20:59.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:00.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:00.105+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:00.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:00.661+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:01.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:01.069+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:01.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:01.653+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:02.075 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:02.073+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:02.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:02.649+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:03.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:03.037+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:03.679 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:03.677+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:04.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:04.021+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:04.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:04.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:05.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:05.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:05.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:05.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:06.047 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:06.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:06.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:06.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:07.090 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:07.085+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:07.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:07.673+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:08.106 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:08.101+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:08.668 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:08.665+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:09.069 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:09.065+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:09.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:09.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:10.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:10.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:10.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:10.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:11.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:11.069+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:11.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:11.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:12.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:12.109+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:12.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:12.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:13.079 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:13.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:13.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:13.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:14.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:14.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:14.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:14.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:15.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:15.113+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:15.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:15.761+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:16.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:16.157+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:16.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:16.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:17.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:17.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:17.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:17.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:18.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:18.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:18.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:18.777+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:19.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:19.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:19.759 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:19.753+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:20.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:20.229+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:20.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:20.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:21.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:21.253+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:21.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:21.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:22.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:22.241+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:22.866 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:22.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:23.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:23.205+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:23.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:23.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:24.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:24.209+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:24.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:24.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:25.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:25.237+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:25.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:25.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:26.201 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:26.197+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:26.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:26.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:27.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:27.165+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:27.820 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:27.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:28.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:28.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:28.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:28.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:29.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:29.205+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:29.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:29.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:30.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:30.205+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:30.902 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:30.897+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:31.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:31.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:31.854 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:31.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:32.207 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:32.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:32.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:32.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:33.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:33.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:33.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:33.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:34.172 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:34.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:34.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:34.896+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:35.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:35.192+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:35.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:35.920+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:36.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:36.216+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:36.879 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:36.876+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:37.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:37.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:37.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:37.904+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:38.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:38.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:38.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:38.872+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:39.272 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:39.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:39.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:39.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:40.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:40.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:40.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:40.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:41.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:41.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:41.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:41.828+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:42.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:42.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:42.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:42.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:43.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:43.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:43.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:43.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:44.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:44.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:44.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:44.772+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:45.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:45.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:45.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:45.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:46.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:46.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:46.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:46.688+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:47.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:47.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:47.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:47.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:48.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:48.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:48.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:48.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:49.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:49.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:49.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:49.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:50.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:50.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:50.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:50.756+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:51.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:51.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:51.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:51.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:52.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:52.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:52.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:52.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:53.241 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:53.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:53.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:53.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:54.247 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:54.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:54.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:54.748+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:55.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:55.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:55.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:55.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:56.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:56.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:56.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:56.824+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:57.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:57.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:57.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:57.832+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:58.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:58.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:58.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:58.836+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:59.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:21:59.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:21:59.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:21:59.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:00.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:00.368+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:00.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:00.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:01.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:01.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:01.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:01.836+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:02.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:02.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:02.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:02.868+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:03.385 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:03.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:03.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:03.880+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:04.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:04.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:04.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:04.868+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:05.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:05.360+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:05.873 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:05.868+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:06.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:06.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:06.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:06.824+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:07.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:07.368+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:07.849 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:07.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:08.418 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:08.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:08.813 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:08.812+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:09.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:09.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:09.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:09.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:10.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:10.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:10.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:10.828+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:11.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:11.372+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:11.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:11.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:12.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:12.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:12.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:12.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:13.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:13.356+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:13.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:13.756+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:14.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:14.320+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:14.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:14.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:15.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:15.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:15.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:15.764+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:16.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:16.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:16.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:16.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:17.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:17.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:17.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:17.748+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:18.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:18.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:18.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:18.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:19.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:19.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:19.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:19.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:20.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:20.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:20.808 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:20.808+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:21.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:21.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:21.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:21.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:22.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:22.296+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:22.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:22.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:23.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:23.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:23.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:23.887+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:24.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:24.279+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:24.932 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:24.931+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:25.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:25.235+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:25.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:25.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:26.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:26.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:26.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:26.967+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:27.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:27.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:27.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:27.919+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:28.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:28.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:28.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:28.903+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:29.146 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:29.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:29.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:29.859+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:30.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:30.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:30.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:30.835+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:31.217 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:31.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:31.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:31.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:32.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:32.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:32.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:32.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:33.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:33.247+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:33.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:33.787+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:34.283 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:34.283+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:34.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:34.791+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:35.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:35.295+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:35.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:35.827+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:36.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:36.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:36.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:36.835+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:37.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:37.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:37.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:37.851+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:38.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:38.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:38.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:38.891+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:39.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:39.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:39.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:39.895+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:40.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:40.243+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:40.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:40.895+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:41.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:41.251+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:41.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:41.871+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:42.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:42.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:42.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:42.847+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:43.337 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:43.335+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:43.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:43.851+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:44.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:44.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:44.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:44.875+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:45.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:45.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:45.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:45.867+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:46.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:46.323+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:46.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:46.855+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:47.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:47.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:47.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:47.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:48.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:48.267+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:48.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:48.827+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:49.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:49.271+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:49.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:49.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:50.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:50.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:50.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:50.831+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:51.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:51.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:51.847 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:51.847+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:52.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:52.275+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:52.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:52.815+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:53.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:53.307+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:53.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:53.831+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:54.335 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:54.335+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:54.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:54.799+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:55.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:55.319+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:55.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:55.799+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:56.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:56.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:56.815 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:56.815+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:57.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:57.283+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:57.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:57.811+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:58.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:58.255+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:58.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:58.771+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:59.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:22:59.271+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:22:59.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:22:59.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:00.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:00.287+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:00.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:00.791+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:01.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:01.303+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:01.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:01.791+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:02.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:02.259+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:02.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:02.795+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:03.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:03.235+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:03.835 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:03.835+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:04.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:04.191+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:04.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:04.799+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:05.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:05.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:05.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:05.775+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:06.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:06.171+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:06.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:06.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:07.133 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:07.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:07.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:07.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:08.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:08.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:08.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:08.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:09.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:09.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:09.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:09.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:10.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:10.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:10.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:10.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:11.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:11.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:11.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:11.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:12.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:12.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:12.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:12.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:13.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:13.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:13.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:13.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:14.190 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:14.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:14.607 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:14.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:15.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:15.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:15.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:15.639+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:16.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:16.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:16.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:16.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:17.212 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:17.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:17.641 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:17.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:18.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:18.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:18.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:18.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:19.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:19.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:19.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:19.682+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:20.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:20.314+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:20.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:20.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:21.307 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:21.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:21.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:21.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:22.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:22.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:22.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:22.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:23.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:23.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:23.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:23.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:24.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:24.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:24.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:24.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:25.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:25.262+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:25.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:25.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:26.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:26.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:26.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:26.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:27.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:27.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:27.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:27.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:28.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:28.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:28.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:28.678+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:29.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:29.286+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:29.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:29.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:30.326 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:30.322+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:30.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:30.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:31.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:31.362+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:31.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:31.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:32.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:32.386+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:32.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:32.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:33.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:33.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:33.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:33.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:34.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:34.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:34.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:34.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:35.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:35.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:35.682 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:35.678+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:36.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:36.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:36.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:36.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:37.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:37.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:37.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:37.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:38.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:38.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:38.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:38.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:39.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:39.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:39.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:39.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:40.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:40.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:40.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:40.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:41.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:41.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:41.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:41.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:42.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:42.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:42.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:42.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:43.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:43.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:43.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:43.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:44.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:44.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:44.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:44.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:45.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:45.450+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:45.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:45.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:46.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:46.426+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:46.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:46.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:47.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:47.386+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:47.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:47.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:48.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:48.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:48.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:48.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:49.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:49.342+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:49.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:49.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:50.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:50.362+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:50.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:50.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:51.367 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:51.366+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:51.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:51.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:52.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:52.350+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:52.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:52.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:53.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:53.302+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:53.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:53.898+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:54.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:54.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:54.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:54.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:55.221 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:55.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:55.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:55.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:56.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:56.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:56.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:56.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:57.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:57.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:57.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:57.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:58.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:58.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:58.845 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:58.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:59.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:23:59.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:23:59.835 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:23:59.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:00.299 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:00.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:00.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:00.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:01.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:01.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:01.813 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:01.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:02.358 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:02.354+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:02.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:02.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:03.373 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:03.370+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:03.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:03.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:04.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:04.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:04.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:04.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:05.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:05.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:05.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:05.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:06.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:06.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:06.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:06.790+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:07.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:07.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:07.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:07.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:08.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:08.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:08.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:08.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:09.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:09.509+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:09.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:09.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:10.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:10.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:10.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:10.777+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:11.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:11.461+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:11.745 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:11.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:12.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:12.465+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:12.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:12.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:13.451 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:13.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:13.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:13.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:14.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:14.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:14.727 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:14.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:15.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:15.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:15.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:15.717+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:16.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:16.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:16.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:16.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:17.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:17.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:17.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:17.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:18.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:18.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:18.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:18.769+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:19.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:19.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:19.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:19.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:20.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:20.557+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:20.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:20.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:21.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:21.597+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:21.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:21.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:22.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:22.637+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:22.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:22.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:23.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:23.601+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:23.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:23.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:24.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:24.573+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:24.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:24.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:25.544 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:25.541+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:25.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:25.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:26.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:26.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:26.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:26.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:27.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:27.565+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:27.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:27.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:28.611 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:28.609+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:28.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:28.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:29.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:29.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:29.834 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:29.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:30.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:30.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:30.866 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:30.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:31.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:31.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:31.854 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:31.849+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:32.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:32.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:32.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:32.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:33.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:33.697+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:33.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:33.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:34.671 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:34.669+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:34.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:34.837+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:35.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:35.673+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:35.813 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:35.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:36.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:36.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:36.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:36.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:37.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:37.609+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:37.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:37.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:38.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:38.625+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:38.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:38.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:39.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:39.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:39.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:39.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:40.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:40.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:40.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:40.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:41.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:41.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:41.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:41.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:42.649 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:42.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:42.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:42.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:43.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:43.621+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:43.895 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:43.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:44.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:44.581+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:44.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:44.933+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:45.605 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:45.601+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:45.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:45.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:46.620 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:46.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:46.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:46.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:47.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:47.605+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:47.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:47.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:48.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:48.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:48.926 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:48.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:49.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:49.621+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:49.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:49.881+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:50.603 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:50.601+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:50.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:50.909+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:51.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:51.621+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:51.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:51.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:52.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:52.657+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:52.907 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:52.905+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:53.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:53.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:53.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:53.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:54.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:54.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:54.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:54.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:55.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:55.613+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:55.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:55.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:56.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:56.613+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:56.810 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:56.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:57.640 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:57.637+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:57.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:57.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:58.650 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:58.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:58.847 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:58.845+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:59.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:24:59.697+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:24:59.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:24:59.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:00.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:00.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:00.899 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:00.896+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:01.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:01.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:01.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:01.944+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:02.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:02.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:02.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:02.900+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:03.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:03.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:03.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:03.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:04.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:04.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:04.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:04.912+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:05.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:05.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:05.902 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:05.896+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:06.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:06.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:06.893 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:06.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:07.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:07.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:07.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:07.860+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:08.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:08.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:08.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:08.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:09.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:09.676+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:09.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:09.864+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:10.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:10.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:10.862 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:10.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:11.681 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:11.676+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:11.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:11.836+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:12.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:12.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:12.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:12.884+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:13.701 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:13.696+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:13.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:13.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:14.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:14.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:14.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:14.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:15.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:15.688+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:15.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:15.932+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:16.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:16.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:16.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:16.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:17.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:17.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:17.949 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:17.944+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:18.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:18.668+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:18.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:18.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:19.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:19.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:20.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:20.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:20.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:20.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:21.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:21.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:21.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:21.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:22.035 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:22.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:22.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:22.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:23.008 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:23.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:23.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:23.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:24.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:24.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:24.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:24.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:25.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:25.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:25.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:25.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:26.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:26.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:26.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:26.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:26.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:26.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:27.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:27.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:28.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:28.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:28.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:28.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:29.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:29.036+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:29.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:29.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:30.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:30.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:30.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:30.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:31.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:31.076+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:31.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:31.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:32.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:32.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:32.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:32.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:33.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:33.080+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:33.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:33.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:34.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:34.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:34.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:34.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:35.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:35.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:35.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:35.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:36.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:36.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:36.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:36.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:37.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:37.196+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:37.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:37.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:38.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:38.204+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:38.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:38.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:39.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:39.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:39.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:39.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:40.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:40.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:40.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:40.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:41.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:41.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:41.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:41.920+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:42.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:42.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:42.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:42.884+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:43.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:43.164+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:43.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:43.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:44.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:44.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:44.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:44.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:45.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:45.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:45.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:45.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:46.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:46.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:46.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:46.888+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:47.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:47.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:47.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:47.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:48.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:48.156+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:48.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:48.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:49.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:49.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:49.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:49.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:50.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:50.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:50.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:50.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:51.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:51.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:51.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:51.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:52.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:52.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:52.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:52.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:53.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:53.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:53.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:53.891+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:54.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:54.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:54.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:54.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:55.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:55.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:55.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:55.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:56.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:56.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:56.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:56.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:57.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:57.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:57.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:57.931+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:58.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:58.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:58.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:58.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:59.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:25:59.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:25:59.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:25:59.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:00.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:00.035+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:00.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:00.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:01.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:01.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:01.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:01.887+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:02.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:02.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:02.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:02.895+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:03.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:03.035+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:03.911 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:03.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:03.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:03.995+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:04.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:04.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:05.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:05.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:05.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:05.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:05.985 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:05.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:06.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:06.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:06.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:06.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:07.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:07.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:07.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:07.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:08.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:08.947+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:08.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:08.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:09.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:09.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:09.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:09.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:10.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:10.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:11.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:11.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:11.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:11.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:11.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:11.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:12.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:12.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:13.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:13.015+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:13.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:13.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:14.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:14.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:14.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:14.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:15.096 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:15.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:15.919 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:15.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:16.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:16.063+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:16.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:16.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:17.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:17.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:17.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:17.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:18.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:18.079+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:18.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:18.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:19.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:19.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:19.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:19.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:20.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:20.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:20.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:20.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:21.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:21.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:21.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:21.771+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:22.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:22.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:22.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:22.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:23.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:23.155+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:23.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:23.795+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:24.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:24.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:24.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:24.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:25.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:25.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:25.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:25.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:26.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:26.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:26.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:26.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:27.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:27.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:27.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:27.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:28.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:28.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:28.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:28.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:29.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:29.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:29.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:29.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:30.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:30.195+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:30.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:30.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:31.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:31.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:31.735 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:31.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:32.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:32.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:32.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:32.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:33.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:33.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:33.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:33.771+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:34.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:34.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:34.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:34.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:35.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:35.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:35.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:35.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:36.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:36.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:36.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:36.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:37.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:37.155+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:37.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:37.719+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:38.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:38.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:38.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:38.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:39.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:39.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:39.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:39.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:40.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:40.095+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:40.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:40.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:41.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:41.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:41.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:41.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:42.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:42.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:42.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:42.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:43.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:43.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:43.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:43.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:44.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:44.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:44.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:44.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:45.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:45.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:45.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:45.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:46.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:46.078+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:46.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:46.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:47.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:47.042+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:47.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:47.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:48.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:48.078+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:48.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:48.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:49.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:49.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:49.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:49.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:50.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:50.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:50.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:50.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:51.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:51.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:51.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:51.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:52.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:52.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:52.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:52.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:53.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:53.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:53.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:53.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:54.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:54.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:54.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:54.802+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:55.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:55.094+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:55.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:55.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:56.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:56.094+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:56.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:56.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:57.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:57.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:57.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:57.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:58.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:58.034+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:58.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:58.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:59.030 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:26:59.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:26:59.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:26:59.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:00.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:00.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:00.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:00.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:00.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:00.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:01.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:01.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:01.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:01.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:02.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:02.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:03.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:03.010+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:03.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:03.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:04.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:04.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:04.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:04.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:04.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:04.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:05.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:05.806+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:05.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:05.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:06.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:06.830+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:06.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:06.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:07.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:07.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:07.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:07.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:08.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:08.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:08.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:08.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:09.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:09.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:10.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:10.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:10.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:10.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:11.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:11.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:11.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:11.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:12.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:12.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:12.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:12.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:13.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:13.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:13.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:13.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:14.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:14.042+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:14.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:14.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:15.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:15.042+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:15.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:15.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:16.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:16.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:16.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:16.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:17.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:17.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:17.718 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:17.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:18.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:18.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:18.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:18.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:19.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:19.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:19.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:19.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:20.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:20.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:20.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:20.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:21.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:21.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:21.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:21.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:22.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:22.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:22.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:22.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:23.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:23.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:23.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:23.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:24.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:24.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:24.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:24.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:25.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:25.186+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:25.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:25.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:26.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:26.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:26.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:26.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:27.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:27.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:27.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:27.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:28.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:28.154+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:28.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:28.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:29.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:29.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:29.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:29.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:30.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:30.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:30.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:30.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:31.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:31.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:31.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:31.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:32.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:32.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:32.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:32.938+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:33.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:33.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:33.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:33.914+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:34.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:34.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:34.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:34.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:35.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:35.178+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:35.982 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:35.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:36.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:36.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:36.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:36.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:37.189 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:37.186+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:37.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:37.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:38.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:38.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:39.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:39.021+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:39.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:39.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:40.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:40.057+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:40.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:40.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:41.031 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:41.029+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:41.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:41.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:42.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:42.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:42.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:42.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:43.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:43.077+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:43.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:43.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:44.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:44.037+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:44.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:44.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:44.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:44.993+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:45.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:45.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:45.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:45.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:46.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:46.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:46.924 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:46.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:47.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:47.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:47.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:47.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:48.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:48.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:48.956 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:48.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:49.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:49.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:49.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:49.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:50.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:50.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:50.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:50.945+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:51.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:51.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:51.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:51.925+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:52.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:52.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:52.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:52.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:53.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:53.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:53.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:53.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:54.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:54.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:54.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:54.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:55.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:55.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:55.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:55.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:56.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:56.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:56.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:56.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:57.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:57.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:57.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:57.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:58.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:58.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:58.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:58.849+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:59.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:27:59.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:27:59.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:27:59.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:00.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:00.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:00.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:00.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:01.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:01.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:01.891 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:01.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:02.231 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:02.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:02.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:02.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:03.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:03.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:03.870 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:03.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:04.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:04.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:04.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:04.849+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:05.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:05.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:05.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:05.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:06.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:06.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:06.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:06.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:07.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:07.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:07.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:07.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:08.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:08.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:08.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:08.845+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:09.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:09.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:09.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:09.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:10.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:10.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:10.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:10.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:11.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:11.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:11.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:11.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:12.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:12.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:12.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:12.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:13.307 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:13.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:13.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:13.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:14.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:14.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:14.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:14.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:15.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:15.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:15.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:15.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:16.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:16.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:16.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:16.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:17.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:17.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:17.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:17.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:18.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:18.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:18.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:18.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:19.250 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:19.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:19.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:19.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:20.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:20.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:20.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:20.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:21.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:21.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:21.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:21.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:22.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:22.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:22.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:22.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:23.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:23.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:23.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:23.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:24.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:24.365+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:24.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:24.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:25.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:25.365+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:25.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:25.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:26.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:26.325+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:26.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:26.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:27.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:27.353+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:27.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:27.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:28.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:28.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:28.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:28.829+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:29.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:29.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:29.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:29.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:30.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:30.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:30.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:30.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:31.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:31.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:31.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:31.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:32.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:32.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:32.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:32.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:33.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:33.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:33.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:33.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:34.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:34.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:34.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:34.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:35.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:35.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:35.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:35.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:36.351 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:36.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:36.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:36.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:37.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:37.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:37.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:37.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:38.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:38.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:38.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:38.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:39.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:39.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:39.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:39.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:40.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:40.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:40.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:40.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:41.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:41.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:41.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:41.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:42.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:42.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:42.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:42.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:43.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:43.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:43.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:43.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:44.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:44.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:44.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:44.888+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:45.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:45.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:45.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:45.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:46.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:46.256+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:46.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:46.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:47.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:47.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:47.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:47.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:48.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:48.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:48.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:48.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:49.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:49.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:49.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:49.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:50.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:50.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:50.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:50.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:51.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:51.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:51.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:51.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:52.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:52.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:52.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:52.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:53.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:53.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:53.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:53.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:54.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:54.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:54.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:54.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:55.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:55.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:55.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:55.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:56.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:56.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:56.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:56.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:57.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:57.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:57.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:57.928+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:58.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:58.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:58.944 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:58.940+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:59.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:28:59.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:28:59.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:28:59.988+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:00.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:00.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:01.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:00.996+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:01.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:01.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:01.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:01.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:02.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:02.424+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:03.027 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:03.020+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:03.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:03.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:04.025 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:04.020+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:04.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:04.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:05.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:05.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:05.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:05.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:06.038 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:06.032+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:06.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:06.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:07.078 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:07.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:07.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:07.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:08.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:08.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:08.439 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:08.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:09.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:09.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:09.455 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:09.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:10.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:10.108+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:10.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:10.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:11.137 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:11.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:11.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:11.504+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:12.152 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:12.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:12.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:12.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:13.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:13.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:13.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:13.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:14.156 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:14.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:14.411 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:14.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:15.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:15.172+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:15.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:15.408+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:16.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:16.164+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:16.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:16.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:17.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:17.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:17.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:17.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:18.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:18.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:18.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:18.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:19.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:19.208+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:19.373 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:19.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:20.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:20.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:20.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:20.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:21.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:21.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:21.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:21.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:22.307 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:22.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:22.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:22.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:23.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:23.351+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:23.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:23.391+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:24.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:24.363+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:24.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:24.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:25.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:25.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:25.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:25.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:26.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:26.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:26.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:26.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:27.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:27.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:27.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:27.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:28.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:28.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:28.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:28.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:29.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:29.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:29.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:29.483+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:30.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:30.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:30.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:30.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:31.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:31.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:31.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:31.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:32.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:32.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:32.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:32.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:33.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:33.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:33.405 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:33.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:34.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:34.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:34.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:34.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:35.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:35.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:35.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:35.407+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:36.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:36.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:36.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:36.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:37.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:37.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:37.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:37.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:38.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:38.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:38.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:38.451+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:39.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:39.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:39.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:39.459+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:40.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:40.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:40.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:40.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:41.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:41.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:41.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:41.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:42.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:42.047+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:42.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:42.567+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:43.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:43.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:43.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:43.567+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:44.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:44.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:44.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:44.527+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:45.083 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:45.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:45.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:45.543+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:46.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:46.067+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:46.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:46.507+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:47.081 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:47.079+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:47.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:47.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:48.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:48.047+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:48.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:48.475+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:49.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:49.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:49.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:49.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:50.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:50.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:50.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:50.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:51.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:51.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:51.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:51.475+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:52.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:52.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:52.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:52.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:53.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:53.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:53.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:53.399+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:54.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:54.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:54.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:54.399+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:55.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:55.027+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:55.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:55.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:55.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:55.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:56.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:56.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:57.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:57.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:57.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:57.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:58.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:58.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:58.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:58.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:59.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:29:59.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:29:59.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:29:59.451+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:00.059 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:00.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:00.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:00.411+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:01.092 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:01.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:01.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:01.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:02.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:02.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:02.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:02.359+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:03.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:03.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:03.396 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:03.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:04.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:04.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:04.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:04.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:05.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:05.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:05.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:05.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:06.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:06.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:06.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:06.411+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:07.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:07.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:07.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:07.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:08.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:08.183+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:08.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:08.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:09.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:09.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:09.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:09.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:10.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:10.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:10.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:10.427+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:11.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:11.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:11.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:11.455+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:12.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:12.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:12.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:12.503+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:13.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:13.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:13.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:13.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:14.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:14.223+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:14.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:14.543+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:15.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:15.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:15.587 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:15.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:16.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:16.302+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:16.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:16.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:17.271 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:17.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:17.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:17.546+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:18.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:18.318+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:18.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:18.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:19.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:19.350+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:19.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:19.566+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:20.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:20.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:20.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:20.566+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:21.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:21.358+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:21.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:21.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:22.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:22.382+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:22.641 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:22.638+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:23.354 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:23.350+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:23.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:23.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:24.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:24.322+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:24.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:24.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:25.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:25.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:25.544 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:25.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:26.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:26.358+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:26.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:26.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:27.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:27.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:27.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:27.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:28.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:28.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:28.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:28.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:29.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:29.330+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:29.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:29.546+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:30.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:30.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:30.501 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:30.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:31.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:31.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:31.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:31.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:32.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:32.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:32.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:32.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:33.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:33.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:33.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:33.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:34.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:34.294+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:34.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:34.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:35.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:35.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:35.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:35.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:36.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:36.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:36.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:36.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:37.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:37.274+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:37.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:37.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:38.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:38.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:38.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:38.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:39.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:39.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:39.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:39.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:40.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:40.154+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:40.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:40.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:41.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:41.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:41.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:41.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:42.138 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:42.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:42.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:42.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:43.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:43.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:43.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:43.438+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:44.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:44.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:44.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:44.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:45.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:45.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:45.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:45.374+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:46.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:46.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:46.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:46.354+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:47.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:47.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:47.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:47.346+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:48.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:48.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:48.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:48.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:49.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:49.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:49.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:49.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:50.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:50.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:50.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:50.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:51.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:51.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:51.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:51.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:52.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:52.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:52.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:52.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:53.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:53.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:53.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:53.326+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:54.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:54.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:54.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:54.374+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:55.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:55.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:55.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:55.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:56.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:56.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:56.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:56.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:57.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:57.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:57.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:57.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:58.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:58.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:58.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:58.434+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:59.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:30:59.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:30:59.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:30:59.446+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:00.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:00.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:00.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:00.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:01.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:01.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:01.422 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:01.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:02.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:02.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:02.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:02.382+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:03.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:03.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:03.337 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:03.334+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:04.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:04.174+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:04.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:04.338+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:05.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:05.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:05.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:05.338+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:06.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:06.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:06.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:06.314+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:07.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:07.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:07.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:07.285+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:08.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:08.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:08.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:08.257+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:09.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:09.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:09.294 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:09.289+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:10.183 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:10.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:10.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:10.253+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:11.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:11.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:11.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:11.213+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:12.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:12.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:12.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:12.245+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:13.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:13.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:13.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:13.233+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:14.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:14.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:14.192 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:14.189+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:15.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:15.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:15.220 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:15.217+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:16.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:16.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:16.258 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:16.253+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:17.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:17.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:17.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:17.269+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:18.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:18.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:18.266 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:18.261+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:19.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:19.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:19.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:19.237+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:20.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:20.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:20.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:20.257+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:21.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:21.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:21.266 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:21.261+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:22.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:22.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:22.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:22.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:23.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:23.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:23.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:23.249+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:24.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:24.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:24.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:24.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:25.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:25.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:25.211 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:25.209+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:26.080 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:26.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:26.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:26.221+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:27.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:27.049+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:27.244 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:27.241+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:28.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:28.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:28.233 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:28.229+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:29.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:29.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:29.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:29.189+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:30.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:30.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:30.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:30.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:31.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:31.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:31.222 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:31.217+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:32.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:32.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:32.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:32.261+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:33.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:33.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:33.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:33.245+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:34.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:34.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:34.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:34.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:35.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:35.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:35.237 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:35.233+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:36.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:36.225+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:36.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:36.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:37.184 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:37.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:37.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:37.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:38.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:38.181+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:38.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:38.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:39.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:39.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:39.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:39.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:40.176 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:40.173+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:40.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:40.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:41.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:41.157+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:41.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:41.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:42.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:42.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:42.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:42.353+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:43.166 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:43.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:43.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:43.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:44.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:44.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:44.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:44.341+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:45.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:45.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:45.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:45.361+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:46.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:46.205+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:46.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:46.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:47.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:47.201+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:47.380 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:47.377+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:48.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:48.217+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:48.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:48.397+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:49.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:49.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:49.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:49.369+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:50.187 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:50.185+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:50.368 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:50.365+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:51.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:51.161+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:51.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:51.381+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:52.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:52.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:52.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:52.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:53.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:53.129+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:53.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:53.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:54.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:54.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:54.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:54.469+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:55.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:55.145+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:55.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:55.433+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:56.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:56.193+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:56.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:56.385+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:57.162 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:57.157+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:57.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:57.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:58.143 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:58.141+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:58.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:58.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:59.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:31:59.177+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:31:59.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:31:59.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:00.185 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:00.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:00.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:00.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:01.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:01.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:01.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:01.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:02.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:02.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:02.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:02.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:03.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:03.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:03.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:03.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:04.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:04.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:04.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:04.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:05.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:05.276+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:05.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:05.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:06.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:06.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:06.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:06.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:07.314 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:07.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:07.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:07.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:08.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:08.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:08.348 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:08.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:09.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:09.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:09.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:09.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:10.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:10.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:10.412 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:10.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:11.240 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:11.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:11.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:11.372+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:12.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:12.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:12.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:12.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:13.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:13.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:13.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:13.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:14.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:14.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:14.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:14.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:15.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:15.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:15.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:15.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:16.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:16.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:16.330 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:16.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:17.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:17.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:17.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:17.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:18.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:18.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:18.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:18.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:19.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:19.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:19.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:19.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:20.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:20.376+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:20.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:20.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:21.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:21.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:21.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:21.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:22.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:22.360+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:22.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:22.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:23.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:23.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:23.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:23.420+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:24.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:24.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:24.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:24.460+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:25.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:25.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:25.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:25.472+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:26.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:26.360+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:26.470 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:26.464+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:27.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:27.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:27.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:27.428+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:28.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:28.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:28.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:28.444+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:29.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:29.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:29.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:29.480+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:30.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:30.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:30.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:30.528+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:31.302 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:31.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:31.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:31.572+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:32.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:32.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:32.534 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:32.528+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:33.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:33.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:33.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:33.484+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:34.371 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:34.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:34.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:34.492+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:35.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:35.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:35.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:35.460+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:36.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:36.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:36.468 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:36.464+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:37.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:37.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:37.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:37.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:38.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:38.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:38.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:38.452+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:39.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:39.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:39.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:39.428+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:40.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:40.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:40.441 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:40.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:41.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:41.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:41.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:41.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:42.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:42.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:42.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:42.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:43.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:43.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:43.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:43.472+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:44.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:44.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:44.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:44.452+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:45.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:45.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:45.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:45.460+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:46.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:46.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:46.418 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:46.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:47.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:47.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:47.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:47.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:48.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:48.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:48.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:48.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:49.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:49.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:49.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:49.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:50.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:50.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:50.414 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:50.412+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:51.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:51.376+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:51.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:51.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:52.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:52.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:52.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:52.407+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:53.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:53.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:53.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:53.451+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:54.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:54.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:54.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:54.487+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:55.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:55.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:55.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:55.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:56.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:56.399+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:56.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:56.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:57.386 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:57.383+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:57.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:57.491+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:58.406 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:58.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:58.541 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:58.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:59.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:32:59.451+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:32:59.549 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:32:59.547+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:00.496 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:00.495+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:00.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:00.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:01.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:01.495+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:01.586 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:01.583+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:02.538 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:02.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:02.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:02.595+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:03.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:03.499+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:03.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:03.643+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:04.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:04.463+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:04.655 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:04.655+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:05.495 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:05.495+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:05.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:05.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:06.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:06.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:06.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:06.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:07.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:07.547+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:07.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:07.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:08.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:08.519+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:08.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:08.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:09.533 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:09.531+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:09.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:09.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:10.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:10.555+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:10.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:10.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:11.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:11.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:11.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:11.691+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:12.526 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:12.523+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:12.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:12.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:13.566 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:13.563+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:13.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:13.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:14.543 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:14.543+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:14.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:14.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:15.590 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:15.587+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:15.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:15.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:16.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:16.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:16.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:16.783+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:17.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:17.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:17.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:17.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:18.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:18.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:18.815 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:18.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:19.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:19.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:19.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:19.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:20.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:20.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:20.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:20.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:21.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:21.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:21.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:21.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:22.673 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:22.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:22.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:22.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:23.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:23.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:23.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:23.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:24.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:24.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:24.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:24.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:25.690 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:25.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:25.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:25.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:26.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:26.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:26.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:26.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:27.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:27.763+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:27.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:27.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:28.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:28.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:28.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:28.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:29.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:29.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:29.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:29.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:30.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:30.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:30.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:30.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:31.663 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:31.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:31.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:31.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:32.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:32.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:32.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:32.915+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:33.677 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:33.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:33.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:33.891+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:34.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:34.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:34.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:34.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:35.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:35.607+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:35.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:35.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:36.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:36.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:36.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:36.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:37.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:37.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:37.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:37.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:38.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:38.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:38.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:38.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:39.680 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:39.679+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:39.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:39.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:40.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:40.671+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:40.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:40.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:41.679 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:41.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:41.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:41.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:42.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:42.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:42.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:42.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:43.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:43.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:43.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:43.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:44.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:44.722+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:44.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:44.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:45.712 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:45.710+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:45.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:45.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:46.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:46.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:46.750 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:46.746+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:47.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:47.722+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:47.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:47.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:48.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:48.730+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:48.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:48.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:49.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:49.686+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:49.732 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:49.730+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:50.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:50.658+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:50.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:50.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:51.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:51.654+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:51.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:51.774+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:52.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:52.662+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:52.815 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:52.814+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:53.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:53.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:53.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:53.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:54.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:54.666+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:54.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:54.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:55.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:55.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:55.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:55.774+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:56.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:56.718+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:56.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:56.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:57.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:57.686+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:57.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:57.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:58.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:58.714+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:58.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:58.782+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:59.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:33:59.686+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:33:59.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:33:59.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:00.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:00.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:00.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:00.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:01.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:01.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:01.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:01.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:02.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:02.690+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:02.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:02.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:03.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:03.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:03.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:03.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:04.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:04.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:04.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:04.870+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:05.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:05.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:05.852 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:05.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:06.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:06.774+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:06.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:06.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:07.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:07.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:07.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:07.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:08.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:08.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:08.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:08.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:09.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:09.730+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:09.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:09.842+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:10.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:10.734+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:10.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:10.802+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:11.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:11.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:11.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:11.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:12.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:12.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:12.807 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:12.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:13.761 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:13.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:13.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:13.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:14.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:14.726+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:14.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:14.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:15.722 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:15.718+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:15.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:15.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:16.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:16.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:16.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:16.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:17.666 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:17.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:17.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:17.730+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:18.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:18.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:18.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:18.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:19.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:19.630+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:19.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:19.730+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:20.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:20.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:20.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:20.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:21.641 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:21.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:21.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:21.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:22.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:22.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:22.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:22.774+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:23.610 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:23.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:23.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:23.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:24.574 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:24.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:24.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:24.694+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:25.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:25.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:25.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:25.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:26.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:26.594+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:26.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:26.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:27.578 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:27.574+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:27.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:27.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:28.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:28.578+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:28.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:28.766+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:29.625 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:29.622+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:29.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:29.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:30.637 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:30.634+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:30.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:30.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:31.639 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:31.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:31.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:31.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:32.676 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:32.674+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:32.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:32.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:33.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:33.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:33.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:33.754+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:34.624 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:34.618+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:34.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:34.802+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:35.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:35.574+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:35.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:35.806+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:36.581 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:36.577+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:36.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:36.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:37.555 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:37.553+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:37.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:37.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:38.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:38.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:38.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:38.865+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:39.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:39.541+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:39.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:39.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:40.548 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:40.545+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:40.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:40.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:41.594 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:41.589+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:41.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:41.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:42.602 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:42.597+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:42.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:42.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:43.596 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:43.593+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:43.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:43.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:44.635 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:44.633+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:44.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:44.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:45.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:45.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:45.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:45.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:46.649 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:46.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:46.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:46.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:47.625 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:47.621+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:47.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:47.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:48.643 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:48.641+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:48.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:48.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:49.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:49.621+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:49.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:49.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:50.632 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:50.629+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:50.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:50.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:51.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:51.641+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:51.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:51.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:52.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:52.689+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:52.806 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:52.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:53.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:53.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:53.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:53.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:54.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:54.709+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:54.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:54.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:55.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:55.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:55.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:55.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:56.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:56.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:56.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:56.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:57.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:57.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:57.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:57.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:58.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:58.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:58.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:58.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:59.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:34:59.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:34:59.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:34:59.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:00.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:00.729+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:00.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:00.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:01.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:01.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:01.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:01.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:02.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:02.769+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:02.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:02.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:03.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:03.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:03.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:03.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:04.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:04.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:04.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:04.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:05.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:05.777+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:05.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:05.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:06.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:06.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:06.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:06.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:07.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:07.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:07.864 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:07.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:08.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:08.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:08.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:08.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:09.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:09.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:09.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:09.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:10.781 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-12T00:35:10.777+0000 7fd5e8fc6700 0 -- 172.21.15.178:0/3737267521 >> [v2:172.21.15.178:6800/20056,v1:172.21.15.178:6801/20056] conn(0x560328bd4930 msgr2=0x7fd5cc0393c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-01-12T00:35:10.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:10.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:10.879 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:10.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:11.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:11.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:11.878 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:11.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:12.836 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:12.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:12.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:12.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:13.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:13.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:13.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:13.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:14.792 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:14.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:14.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:14.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:15.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:15.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:15.824 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:15.821+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:16.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:16.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:16.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:16.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:17.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:17.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:17.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:17.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:18.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:18.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:18.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:18.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:19.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:19.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:19.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:19.757+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:20.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:20.705+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:20.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:20.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:21.685 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:21.681+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:21.807 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:21.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:22.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:22.713+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:22.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:22.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:23.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:23.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:23.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:23.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:24.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:24.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:24.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:24.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:25.713 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:25.709+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:25.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:25.877+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:26.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:26.733+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:26.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:26.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:27.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:27.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:27.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:27.889+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:28.765 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:28.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:28.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:28.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:29.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:29.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:29.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:29.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:30.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:30.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:30.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:30.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:31.802 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:31.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:31.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:31.936+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:32.824 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:32.820+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:32.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:32.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:33.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:33.772+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:33.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:33.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:34.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:34.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:34.918 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:34.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:35.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:35.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:35.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:35.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:36.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:36.812+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:36.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:36.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:37.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:37.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:37.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:37.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:38.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:38.764+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:38.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:38.816+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:39.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:39.736+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:39.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:39.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:40.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:40.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:40.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:40.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:41.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:41.720+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:41.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:41.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:42.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:42.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:42.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:42.772+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:43.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:43.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:43.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:43.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:44.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:44.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:44.751 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:44.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:45.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:45.736+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:45.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:45.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:46.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:46.736+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:46.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:46.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:47.708 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:47.704+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:47.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:47.836+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:48.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:48.720+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:48.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:48.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:49.723 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:49.716+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:49.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:49.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:50.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:50.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:50.745 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:50.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:51.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:51.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:51.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:51.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:52.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:52.756+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:52.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:52.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:53.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:53.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:53.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:53.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:54.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:54.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:54.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:54.768+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:55.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:55.768+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:55.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:55.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:56.733 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:56.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:56.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:56.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:57.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:57.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:57.805 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:57.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:58.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:58.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:58.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:58.848+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:59.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:35:59.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:35:59.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:35:59.824+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:00.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:00.688+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:00.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:00.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:01.653 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:01.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:01.900 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:01.896+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:02.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:02.696+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:02.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:02.900+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:03.669 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:03.664+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:03.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:03.920+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:04.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:04.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:04.962 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:04.956+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:05.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:05.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:05.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:05.956+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:06.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:06.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:06.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:06.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:07.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:07.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:08.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:08.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:08.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:08.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:08.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:08.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:09.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:09.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:10.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:10.012+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:10.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:10.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:10.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:10.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:11.745 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:11.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:12.002 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:12.000+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:12.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:12.796+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:13.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:13.020+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:13.761 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:13.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:13.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:13.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:14.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:14.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:15.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:15.020+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:15.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:15.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:16.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:16.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:16.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:16.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:17.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:17.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:17.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:17.800+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:18.007 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:18.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:18.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:18.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:18.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:18.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:19.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:19.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:19.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:19.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:20.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:20.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:20.984 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:20.983+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:21.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:21.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:21.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:21.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:22.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:22.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:23.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:23.007+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:23.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:23.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:24.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:24.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:24.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:24.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:25.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:25.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:25.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:25.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:26.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:26.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:26.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:26.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:27.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:27.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:27.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:27.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:28.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:28.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:28.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:28.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:29.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:29.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:29.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:29.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:30.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:30.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:30.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:30.723+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:31.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:31.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:31.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:31.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:32.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:32.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:32.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:32.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:33.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:33.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:33.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:33.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:34.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:34.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:34.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:34.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:35.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:35.315+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:35.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:35.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:36.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:36.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:36.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:36.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:37.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:37.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:37.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:37.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:38.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:38.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:38.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:38.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:39.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:39.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:39.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:39.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:40.320 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:40.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:40.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:40.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:41.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:41.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:41.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:41.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:42.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:42.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:42.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:42.783+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:43.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:43.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:43.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:43.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:44.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:44.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:44.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:44.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:45.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:45.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:45.826 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:45.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:46.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:46.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:46.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:46.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:47.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:47.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:47.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:47.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:48.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:48.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:48.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:48.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:49.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:49.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:49.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:49.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:50.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:50.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:50.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:50.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:51.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:51.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:51.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:51.795+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:52.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:52.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:52.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:52.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:53.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:53.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:53.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:53.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:54.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:54.271+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:54.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:54.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:55.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:55.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:55.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:55.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:56.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:56.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:56.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:56.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:57.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:57.335+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:57.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:57.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:58.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:58.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:58.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:58.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:59.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:36:59.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:36:59.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:36:59.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:00.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:00.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:00.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:00.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:01.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:01.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:01.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:01.795+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:02.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:02.315+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:02.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:02.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:03.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:03.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:03.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:03.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:04.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:04.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:04.738 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:04.735+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:05.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:05.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:05.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:05.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:06.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:06.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:06.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:06.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:07.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:07.423+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:07.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:07.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:08.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:08.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:08.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:08.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:09.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:09.419+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:09.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:09.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:10.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:10.407+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:10.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:10.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:11.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:11.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:11.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:11.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:12.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:12.451+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:12.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:12.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:13.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:13.434+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:13.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:13.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:14.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:14.390+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:14.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:14.774+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:15.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:15.390+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:15.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:15.730+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:16.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:16.370+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:16.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:16.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:17.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:17.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:17.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:17.686+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:18.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:18.374+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:18.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:18.706+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:19.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:19.370+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:19.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:19.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:20.421 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:20.418+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:20.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:20.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:21.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:21.410+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:21.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:21.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:22.461 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:22.458+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:22.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:22.710+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:23.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:23.466+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:23.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:23.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:24.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:24.438+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:24.801 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:24.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:25.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:25.450+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:25.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:25.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:26.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:26.486+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:26.838 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:26.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:27.515 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:27.514+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:27.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:27.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:28.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:28.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:28.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:28.806+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:29.512 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:29.510+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:29.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:29.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:30.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:30.506+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:30.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:30.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:31.556 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:31.554+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:31.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:31.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:32.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:32.590+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:32.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:32.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:33.582 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:33.578+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:33.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:33.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:34.554 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:34.550+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:34.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:34.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:35.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:35.558+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:35.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:35.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:36.609 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:36.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:36.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:36.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:37.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:37.582+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:37.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:37.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:38.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:38.542+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:38.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:38.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:39.529 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:39.526+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:39.907 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:39.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:40.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:40.502+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:40.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:40.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:41.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:41.478+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:41.909 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:41.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:42.471 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:42.470+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:42.868 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:42.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:43.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:43.482+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:43.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:43.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:44.472 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:44.470+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:44.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:44.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:45.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:45.426+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:45.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:45.798+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:46.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:46.386+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:46.837 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:46.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:47.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:47.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:47.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:47.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:48.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:48.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:48.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:48.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:49.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:49.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:49.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:49.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:50.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:50.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:50.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:50.918+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:51.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:51.322+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:51.890 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:51.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:52.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:52.350+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:52.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:52.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:53.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:53.366+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:53.922 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:53.918+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:54.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:54.402+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:54.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:54.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:55.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:55.422+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:55.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:55.934+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:56.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:56.426+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:56.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:56.966+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:57.447 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:57.446+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:57.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:57.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:58.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:58.414+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:58.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:58.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:59.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:37:59.438+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:37:59.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:37:59.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:00.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:00.418+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:00.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:00.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:01.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:01.390+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:01.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:01.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:02.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:02.442+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:02.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:02.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:03.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:03.430+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:03.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:03.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:04.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:04.482+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:04.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:04.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:05.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:05.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:05.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:05.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:06.416 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:06.413+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:06.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:06.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:07.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:07.401+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:07.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:07.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:08.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:08.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:08.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:08.801+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:09.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:09.413+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:09.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:09.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:10.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:10.405+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:10.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:10.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:11.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:11.433+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:11.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:11.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:12.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:12.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:12.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:12.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:13.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:13.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:13.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:13.845+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:14.469 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:14.465+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:14.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:14.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:15.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:15.425+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:15.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:15.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:16.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:16.417+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:16.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:16.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:17.431 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:17.429+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:17.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:17.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:18.443 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:18.441+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:18.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:18.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:19.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:19.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:19.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:19.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:20.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:20.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:20.778 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:20.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:21.464 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:21.461+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:21.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:21.809+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:22.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:22.437+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:22.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:22.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:23.480 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:23.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:23.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:23.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:24.482 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:24.477+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:24.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:24.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:25.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:25.433+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:25.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:25.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:26.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:26.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:26.800 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:26.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:27.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:27.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:27.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:27.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:28.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:28.489+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:28.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:28.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:29.515 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:29.513+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:29.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:29.857+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:30.498 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:30.493+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:30.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:30.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:31.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:31.509+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:31.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:31.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:32.508 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:32.505+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:32.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:32.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:33.514 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:33.509+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:33.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:33.849+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:34.542 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:34.537+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:34.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:34.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:35.504 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:35.501+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:35.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:35.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:36.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:36.509+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:36.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:36.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:37.492 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:37.489+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:37.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:37.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:38.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:38.497+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:38.807 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:38.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:39.452 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:39.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:39.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:39.793+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:40.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:40.449+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:40.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:40.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:41.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:41.457+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:41.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:41.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:42.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:42.469+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:42.737 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:42.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:43.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:43.505+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:43.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:43.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:44.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:44.541+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:44.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:44.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:45.531 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:45.529+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:45.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:45.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:46.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:46.505+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:46.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:46.701+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:47.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:47.465+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:47.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:47.717+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:48.470 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:48.465+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:48.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:48.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:49.475 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:49.473+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:49.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:49.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:50.450 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:50.445+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:50.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:50.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:51.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:51.453+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:51.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:51.733+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:52.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:52.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:52.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:52.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:53.525 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:53.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:53.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:53.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:54.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:54.549+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:54.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:54.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:55.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:55.541+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:55.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:55.797+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:56.520 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:56.517+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:56.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:56.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:57.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:57.485+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:57.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:57.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:58.467 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:58.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:58.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:58.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:59.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:38:59.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:38:59.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:38:59.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:00.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:00.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:00.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:00.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:01.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:01.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:01.718 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:01.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:02.412 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:02.408+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:02.691 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:02.688+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:03.441 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:03.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:03.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:03.656+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:04.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:04.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:04.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:04.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:05.391 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:05.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:05.649 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:05.644+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:06.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:06.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:06.664 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:06.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:07.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:07.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:07.627 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:07.624+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:08.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:08.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:08.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:08.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:09.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:09.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:09.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:09.596+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:10.410 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:10.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:10.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:10.576+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:11.437 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:11.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:11.554 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:11.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:12.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:12.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:12.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:12.516+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:13.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:13.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:13.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:13.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:14.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:14.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:14.601 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:14.596+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:15.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:15.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:15.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:15.636+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:16.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:16.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:16.673 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:16.668+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:17.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:17.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:17.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:17.696+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:18.396 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:18.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:18.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:18.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:19.387 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:19.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:19.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:19.676+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:20.430 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:20.424+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:20.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:20.676+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:21.468 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:21.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:21.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:21.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:22.423 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:22.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:22.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:22.652+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:23.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:23.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:23.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:23.648+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:24.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:24.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:24.641 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:24.636+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:25.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:25.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:25.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:25.608+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:26.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:26.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:26.599 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:26.596+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:27.518 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:27.512+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:27.615 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:27.608+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:28.522 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:28.516+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:28.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:28.608+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:29.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:29.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:29.623 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:29.620+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:30.530 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:30.524+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:30.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:30.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:31.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:31.544+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:31.666 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:31.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:32.532 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:32.528+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:32.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:32.624+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:33.511 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:33.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:33.586 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:33.580+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:34.474 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:34.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:34.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:34.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:35.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:35.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:35.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:35.608+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:36.458 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:36.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:36.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:36.572+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:37.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:37.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:37.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:37.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:38.401 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:38.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:38.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:38.592+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:39.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:39.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:39.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:39.576+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:40.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:40.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:40.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:40.552+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:41.372 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:41.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:41.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:41.556+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:42.379 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:42.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:42.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:42.512+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:43.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:43.376+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:43.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:43.476+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:44.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:44.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:44.527 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:44.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:45.296 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:45.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:45.500 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:45.500+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:46.336 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:46.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:46.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:46.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:47.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:47.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:47.506 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:47.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:48.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:48.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:48.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:48.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:49.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:49.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:49.566 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:49.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:50.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:50.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:50.588 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:50.588+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:51.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:51.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:51.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:51.611+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:52.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:52.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:52.601 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:52.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:53.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:53.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:53.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:53.635+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:54.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:54.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:54.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:54.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:55.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:55.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:55.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:55.619+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:56.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:56.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:56.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:56.627+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:57.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:57.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:57.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:57.607+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:58.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:58.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:58.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:58.583+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:59.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:39:59.367+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:39:59.547 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:39:59.547+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:00.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:00.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:00.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:00.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:01.280 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:01.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:01.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:01.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:02.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:02.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:02.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:02.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:03.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:03.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:03.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:03.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:04.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:04.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:04.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:04.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:05.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:05.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:05.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:05.543+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:06.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:06.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:06.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:06.543+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:07.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:07.155+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:07.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:07.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:08.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:08.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:08.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:08.507+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:09.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:09.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:09.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:09.471+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:10.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:10.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:10.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:10.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:11.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:11.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:11.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:11.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:12.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:12.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:12.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:12.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:13.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:13.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:13.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:13.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:14.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:14.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:14.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:14.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:15.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:15.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:15.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:15.503+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:16.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:16.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:16.459 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:16.459+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:17.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:17.195+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:17.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:17.471+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:18.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:18.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:18.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:18.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:19.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:19.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:19.554 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:19.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:20.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:20.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:20.516 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:20.515+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:21.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:21.271+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:21.554 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:21.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:22.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:22.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:22.522 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:22.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:23.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:23.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:23.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:23.483+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:24.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:24.283+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:24.476 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:24.475+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:25.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:25.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:25.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:25.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:26.335 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:26.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:26.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:26.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:27.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:27.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:27.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:27.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:28.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:28.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:28.542 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:28.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:29.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:29.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:29.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:29.539+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:30.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:30.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:30.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:30.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:31.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:31.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:31.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:31.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:32.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:32.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:32.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:32.527+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:33.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:33.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:33.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:33.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:34.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:34.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:34.552 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:34.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:35.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:35.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:35.520 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:35.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:36.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:36.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:36.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:36.515+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:37.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:37.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:37.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:37.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:38.229 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:38.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:38.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:38.507+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:39.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:39.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:39.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:39.515+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:40.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:40.223+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:40.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:40.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:41.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:41.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:41.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:41.531+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:42.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:42.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:42.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:42.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:43.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:43.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:43.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:43.531+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:44.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:44.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:44.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:44.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:45.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:45.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:45.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:45.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:46.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:46.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:46.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:46.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:47.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:47.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:47.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:47.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:48.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:48.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:48.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:48.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:49.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:49.130+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:49.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:49.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:50.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:50.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:50.516 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:50.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:51.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:51.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:51.528 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:51.526+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:52.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:52.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:52.523 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:52.522+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:53.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:53.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:53.546 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:53.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:54.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:54.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:54.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:54.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:55.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:55.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:55.601 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:55.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:56.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:56.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:56.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:56.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:57.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:57.154+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:57.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:57.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:58.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:58.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:58.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:58.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:59.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:40:59.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:40:59.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:40:59.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:00.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:00.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:00.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:00.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:01.247 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:01.246+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:01.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:01.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:02.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:02.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:02.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:02.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:03.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:03.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:03.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:03.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:04.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:04.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:04.563 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:04.562+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:05.272 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:05.270+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:05.592 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:05.590+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:06.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:06.314+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:06.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:06.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:07.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:07.350+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:07.580 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:07.578+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:08.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:08.398+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:08.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:08.618+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:09.366 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:09.362+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:09.653 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:09.650+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:10.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:10.382+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:10.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:10.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:11.376 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:11.374+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:11.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:11.658+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:12.343 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:12.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:12.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:12.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:13.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:13.326+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:13.602 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:13.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:14.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:14.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:14.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:14.626+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:15.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:15.318+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:15.653 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:15.650+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:16.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:16.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:16.634 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:16.630+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:17.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:17.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:17.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:17.610+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:18.303 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:18.302+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:18.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:18.658+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:19.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:19.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:19.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:19.630+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:20.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:20.234+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:20.675 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:20.674+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:21.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:21.274+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:21.649 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:21.646+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:22.313 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:22.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:22.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:22.686+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:23.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:23.286+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:23.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:23.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:24.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:24.302+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:24.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:24.658+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:25.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:25.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:25.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:25.614+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:26.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:26.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:26.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:26.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:27.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:27.202+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:27.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:27.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:28.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:28.202+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:28.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:28.614+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:29.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:29.178+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:29.586 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:29.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:30.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:30.178+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:30.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:30.626+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:31.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:31.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:31.658 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:31.654+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:32.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:32.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:32.618 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:32.614+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:33.195 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:33.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:33.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:33.638+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:34.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:34.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:34.616 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:34.614+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:35.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:35.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:35.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:35.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:36.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:36.234+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:36.601 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:36.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:37.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:37.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:37.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:37.625+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:38.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:38.205+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:38.625 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:38.621+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:39.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:39.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:39.606 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:39.601+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:40.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:40.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:40.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:40.637+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:41.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:41.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:41.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:41.633+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:42.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:42.225+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:42.591 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:42.589+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:43.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:43.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:43.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:43.557+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:44.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:44.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:44.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:44.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:45.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:45.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:45.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:45.521+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:46.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:46.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:46.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:46.541+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:47.317 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:47.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:47.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:47.533+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:48.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:48.309+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:48.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:48.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:49.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:49.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:49.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:49.461+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:50.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:50.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:50.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:50.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:51.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:51.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:51.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:51.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:52.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:52.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:52.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:52.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:53.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:53.297+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:53.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:53.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:54.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:54.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:54.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:54.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:55.285 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:55.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:55.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:55.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:56.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:56.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:56.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:56.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:57.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:57.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:57.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:57.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:58.216 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:58.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:58.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:58.409+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:59.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:41:59.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:41:59.458 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:41:59.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:00.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:00.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:00.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:00.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:01.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:01.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:01.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:01.409+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:02.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:02.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:02.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:02.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:03.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:03.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:03.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:03.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:04.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:04.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:04.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:04.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:05.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:05.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:05.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:05.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:06.144 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:06.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:06.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:06.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:07.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:07.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:07.328 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:07.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:08.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:08.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:08.324 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:08.321+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:09.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:09.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:09.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:09.273+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:10.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:10.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:10.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:10.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:11.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:11.081+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:11.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:11.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:12.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:12.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:12.335 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:12.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:13.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:13.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:13.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:13.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:14.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:14.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:14.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:14.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:15.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:15.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:15.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:15.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:16.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:16.181+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:16.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:16.381+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:17.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:17.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:17.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:17.373+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:18.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:18.253+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:18.381 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:18.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:19.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:19.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:19.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:19.409+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:20.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:20.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:20.449 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:20.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:21.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:21.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:21.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:21.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:22.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:22.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:22.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:22.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:23.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:23.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:23.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:23.417+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:24.325 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:24.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:24.380 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:24.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:25.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:25.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:25.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:25.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:26.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:26.325+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:26.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:26.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:27.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:27.313+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:27.353 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:27.349+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:28.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:28.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:28.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:28.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:29.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:29.289+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:29.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:29.361+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:30.251 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:30.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:30.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:30.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:31.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:31.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:31.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:31.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:32.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:32.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:32.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:32.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:33.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:33.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:33.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:33.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:34.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:34.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:34.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:34.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:35.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:35.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:35.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:35.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:36.215 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:36.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:36.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:36.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:37.196 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:37.192+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:37.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:37.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:38.206 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:38.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:38.361 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:38.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:39.235 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:39.228+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:39.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:39.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:40.266 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:40.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:40.408 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:40.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:41.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:41.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:41.444 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:41.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:42.181 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:42.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:42.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:42.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:43.161 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:43.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:43.385 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:43.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:44.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:44.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:44.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:44.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:45.160 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:45.156+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:45.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:45.340+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:46.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:46.176+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:46.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:46.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:47.141 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:47.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:47.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:47.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:48.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:48.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:48.318 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:48.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:49.188 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:49.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:49.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:49.336+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:50.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:50.204+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:50.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:50.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:51.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:51.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:51.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:51.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:52.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:52.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:52.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:52.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:53.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:53.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:53.362 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:53.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:54.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:54.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:54.413 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:54.408+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:55.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:55.124+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:55.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:55.384+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:56.084 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:56.080+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:56.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:56.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:57.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:57.068+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:57.373 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:57.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:58.110 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:58.104+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:58.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:58.356+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:59.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:42:59.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:42:59.375 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:42:59.368+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:00.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:00.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:00.424 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:00.420+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:01.097 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:01.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:01.386 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:01.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:02.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:02.104+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:02.367 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:02.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:03.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:03.148+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:03.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:03.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:04.149 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:04.144+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:04.403 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:04.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:05.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:05.136+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:05.432 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:05.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:06.108 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:06.104+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:06.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:06.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:07.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:07.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:07.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:07.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:08.115 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:08.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:08.456 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:08.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:09.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:09.084+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:09.442 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:09.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:10.048 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:10.048+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:10.476 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:10.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:11.089 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:11.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:11.468 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:11.468+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:12.095 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:12.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:12.513 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:12.512+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:13.065 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:13.064+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:13.499 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:13.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:14.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:14.112+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:14.457 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:14.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:15.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:15.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:15.427 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:15.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:16.202 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:16.200+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:16.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:16.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:17.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:17.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:17.445 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:17.444+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:18.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:18.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:18.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:18.492+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:19.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:19.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:19.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:19.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:20.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:20.292+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:20.510 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:20.508+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:21.268 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:21.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:21.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:21.560+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:22.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:22.304+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:22.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:22.548+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:23.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:23.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:23.537 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:23.535+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:24.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:24.335+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:24.503 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:24.503+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:25.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:25.355+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:25.506 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:25.503+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:26.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:26.343+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:26.476 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:26.475+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:27.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:27.359+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:27.517 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:27.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:28.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:28.323+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:28.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:28.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:29.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:29.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:29.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:29.475+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:30.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:30.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:30.515 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:30.515+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:31.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:31.359+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:31.544 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:31.543+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:32.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:32.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:32.506 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:32.503+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:33.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:33.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:33.486 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:33.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:34.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:34.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:34.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:34.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:35.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:35.411+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:35.460 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:35.459+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:36.427 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:36.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:36.494 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:36.491+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:37.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:37.395+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:37.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:37.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:38.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:38.359+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:38.425 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:38.423+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:39.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:39.383+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:39.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:39.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:40.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:40.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:40.375 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:40.375+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:41.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:41.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:41.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:41.415+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:42.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:42.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:42.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:42.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:43.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:43.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:43.388 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:43.387+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:44.345 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:44.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:44.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:44.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:45.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:45.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:45.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:45.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:46.384 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:46.383+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:46.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:46.403+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:47.366 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:47.363+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:47.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:47.431+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:48.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:48.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:48.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:48.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:49.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:49.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:49.466 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:49.463+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:50.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:50.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:50.506 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:50.503+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:51.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:51.375+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:51.521 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:51.519+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:52.383 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:52.383+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:52.572 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:52.571+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:53.355 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:53.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:53.567 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:53.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:54.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:54.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:54.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:54.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:55.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:55.351+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:55.490 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:55.487+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:56.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:56.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:56.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:56.507+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:57.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:57.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:57.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:57.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:58.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:58.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:58.485 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:58.483+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:59.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:43:59.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:43:59.535 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:43:59.535+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:00.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:00.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:00.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:00.551+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:01.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:01.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:01.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:01.603+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:02.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:02.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:02.641 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:02.639+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:03.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:03.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:03.656 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:03.655+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:04.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:04.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:04.683 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:04.683+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:05.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:05.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:05.641 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:05.639+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:06.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:06.183+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:06.651 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:06.651+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:07.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:07.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:07.617 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:07.615+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:08.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:08.127+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:08.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:08.655+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:09.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:09.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:09.614 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:09.611+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:10.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:10.135+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:10.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:10.571+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:11.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:11.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:11.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:11.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:12.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:12.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:12.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:12.567+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:13.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:13.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:13.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:13.575+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:14.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:14.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:14.604 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:14.603+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:15.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:15.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:15.587 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:15.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:16.065 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:16.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:16.544 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:16.542+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:17.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:17.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:17.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:17.570+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:18.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:18.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:18.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:18.582+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:19.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:19.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:19.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:19.574+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:20.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:20.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:20.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:20.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:21.076 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:21.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:21.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:21.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:22.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:22.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:22.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:22.554+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:23.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:22.998+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:23.570 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:23.566+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:23.999 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:23.998+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:24.580 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:24.578+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:24.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:24.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:25.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:25.626+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:25.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:25.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:26.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:26.618+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:26.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:26.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:27.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:27.610+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:27.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:27.950+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:28.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:28.650+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:28.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:28.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:29.609 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:29.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:29.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:29.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:30.561 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:30.558+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:30.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:30.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:31.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:31.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:31.908 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:31.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:32.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:32.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:32.889 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:32.886+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:33.502 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:33.498+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:33.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:33.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:34.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:34.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:34.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:34.822+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:35.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:35.514+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:35.814 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:35.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:36.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:36.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:36.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:36.862+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:37.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:37.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:37.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:37.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:38.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:38.470+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:38.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:38.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:39.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:39.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:39.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:39.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:40.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:40.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:40.833 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:40.830+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:41.494 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:41.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:41.821 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:41.818+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:42.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:42.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:42.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:42.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:43.569 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:43.566+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:43.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:43.858+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:44.536 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:44.534+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:44.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:44.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:45.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:45.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:45.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:45.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:46.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:46.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:46.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:46.846+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:47.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:47.482+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:47.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:47.834+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:48.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:48.466+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:48.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:48.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:49.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:49.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:49.893 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:49.890+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:50.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:50.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:50.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:50.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:51.504 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:51.502+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:51.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:51.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:52.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:52.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:52.875 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:52.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:53.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:53.474+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:53.883 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:53.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:54.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:54.454+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:54.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:54.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:55.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:55.438+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:55.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:55.826+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:56.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:56.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:56.842 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:56.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:57.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:57.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:57.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:57.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:58.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:58.530+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:58.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:58.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:59.507 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:44:59.506+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:44:59.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:44:59.742+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:00.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:00.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:00.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:00.758+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:01.553 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:01.550+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:01.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:01.798+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:02.602 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:02.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:02.839 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:02.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:03.593 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:03.590+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:03.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:03.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:04.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:04.622+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:04.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:04.806+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:05.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:05.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:05.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:05.778+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:06.636 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:06.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:06.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:06.766+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:07.633 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:07.630+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:07.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:07.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:08.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:08.673+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:08.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:08.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:09.659 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:09.657+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:09.812 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:09.809+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:10.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:10.693+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:10.796 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:10.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:11.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:11.681+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:11.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:11.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:12.640 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:12.637+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:12.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:12.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:13.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:13.657+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:13.872 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:13.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:14.622 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:14.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:14.830 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:14.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:15.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:15.585+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:15.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:15.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:16.582 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:16.577+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:16.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:16.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:17.016 INFO:tasks.workunit.client.0.smithi178.stderr:2024-01-12T00:45:17.013+0000 7fd5d57fa700 -1 received signal: Terminated from timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh (PID: 24342) UID: 1000 2024-01-12T00:45:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-12T00:45:17.540 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:17.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:17.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:17.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:18.538 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:18.533+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:18.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:18.797+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:19.526 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:19.521+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:19.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:19.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:20.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:20.525+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:20.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:20.733+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:21.568 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:21.565+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:21.773 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:21.769+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:22.555 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:22.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:22.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:22.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:23.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:23.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:23.769 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:23.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:24.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:24.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:24.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:24.777+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:25.480 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:25.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:25.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:25.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:26.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:26.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:26.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:26.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:27.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:27.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:27.803 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:27.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:28.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:28.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:28.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:28.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:29.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:29.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:29.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:29.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:30.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:30.505+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:30.844 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:30.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:31.518 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:31.513+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:31.816 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:31.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:32.504 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:32.501+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:32.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:32.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:33.541 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:33.537+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:33.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:33.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:34.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:34.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:34.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:34.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:35.530 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:35.525+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:35.867 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:35.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:36.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:36.529+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:36.838 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:36.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:37.489 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:37.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:37.847 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:37.845+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:38.484 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:38.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:38.880 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:38.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:39.506 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:39.501+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:39.915 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:39.913+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:40.511 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:40.509+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:40.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:40.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:41.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:41.529+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:41.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:41.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:42.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:42.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:42.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:42.977+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:43.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:43.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:43.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:43.993+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:44.538 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:44.533+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:45.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:45.021+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:45.576 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:45.573+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:46.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:46.069+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:46.597 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:46.593+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:47.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:47.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:47.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:47.625+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:48.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:48.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:48.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:48.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:49.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:49.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:49.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:49.681+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:50.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:50.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:50.644 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:50.641+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:51.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:51.117+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:51.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:51.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:52.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:52.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:52.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:52.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:53.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:53.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:53.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:53.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:54.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:54.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:54.577 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:54.573+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:55.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:55.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:55.544 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:55.541+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:56.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:56.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:56.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:56.509+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:57.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:57.161+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:57.558 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:57.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:58.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:58.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:58.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:58.561+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:59.155 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:45:59.149+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:45:59.556 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:45:59.553+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:00.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:00.153+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:00.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:00.557+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:01.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:01.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:01.578 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:01.572+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:02.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:02.100+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:02.584 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:02.580+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:03.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:03.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:03.630 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:03.624+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:04.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:04.160+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:04.625 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:04.620+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:05.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:05.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:05.669 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:05.664+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:06.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:06.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:06.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:06.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:07.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:07.164+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:07.736 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:07.732+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:08.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:08.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:08.703 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:08.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:09.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:09.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:09.723 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:09.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:10.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:10.224+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:10.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:10.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:11.263 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:11.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:11.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:11.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:12.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:12.220+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:12.722 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:12.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:13.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:13.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:13.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:13.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:14.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:14.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:14.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:14.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:15.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:15.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:15.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:15.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:16.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:16.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:16.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:16.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:17.018 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-01-12T00:46:17.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:17.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:17.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:17.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:18.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:18.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:18.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:18.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:19.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:19.116+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:19.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:19.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:20.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:20.100+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:20.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:20.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:21.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:21.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:21.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:21.848+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:22.156 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:22.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:22.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:22.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:23.140 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:23.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:23.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:23.868+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:24.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:24.088+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:24.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:24.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:25.070 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:25.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:25.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:25.864+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:26.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:26.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:26.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:26.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:27.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:27.164+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:27.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:27.872+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:28.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:28.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:28.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:28.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:29.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:29.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:29.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:29.916+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:30.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:30.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:30.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:30.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:31.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:31.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:31.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:31.892+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:32.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:32.144+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:32.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:32.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:33.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:33.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:33.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:33.836+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:34.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:34.160+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:34.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:34.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:35.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:35.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:35.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:35.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:36.158 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:36.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:36.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:36.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:37.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:37.156+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:37.910 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:37.904+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:38.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:38.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:38.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:38.912+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:39.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:39.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:39.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:39.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:40.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:40.216+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:40.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:40.968+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:41.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:41.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:41.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:41.980+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:42.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:42.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:43.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:43.024+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:43.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:43.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:44.035 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:44.036+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:44.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:44.192+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:44.992 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:44.992+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:45.186 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:45.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:45.948 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:45.948+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:46.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:46.224+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:46.978 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:46.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:47.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:47.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:48.010 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:48.008+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:48.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:48.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:49.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:49.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:49.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:49.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:50.052 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:50.052+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:50.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:50.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:51.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:51.072+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:51.288 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:51.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:52.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:52.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:52.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:52.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:53.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:53.044+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:53.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:53.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:54.050 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:54.047+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:54.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:54.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:55.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:55.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:55.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:55.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:56.046 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:56.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:56.334 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:56.331+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:57.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:57.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:57.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:57.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:58.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:58.055+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:58.261 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:58.259+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:59.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:46:59.055+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:46:59.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:46:59.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:00.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:00.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:00.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:00.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:01.100 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:01.099+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:01.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:01.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:02.116 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:02.115+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:02.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:02.271+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:03.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:03.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:03.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:03.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:04.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:04.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:04.228 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:04.227+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:05.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:05.187+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:05.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:05.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:06.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:06.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:06.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:06.187+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:07.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:07.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:07.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:07.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:08.170 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:08.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:08.208 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:08.207+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:09.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:09.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:09.177 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:09.175+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:10.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:10.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:10.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:10.183+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:11.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:11.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:11.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:11.199+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:12.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:12.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:12.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:12.227+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:13.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:13.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:13.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:13.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:14.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:14.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:14.204 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:14.203+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:15.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:15.123+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:15.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:15.239+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:16.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:16.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:16.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:16.215+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:17.020 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-01-12T00:47:17.020 INFO:tasks.workunit:Stopping ['rbd/cli_generic.sh'] on client.0... 2024-01-12T00:47:17.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-01-12T00:47:17.180 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:17.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:17.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:17.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:17.314 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/cli_generic.sh) on smithi178 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=49908e15c7044e78475683aa6ccd77bd187be2f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' 2024-01-12T00:47:17.543 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=77c422986b824c2b950f161693dd0696 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/cli_generic.sh) on smithi178 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=49908e15c7044e78475683aa6ccd77bd187be2f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' 2024-01-12T00:47:17.546 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-12T00:47:17.556 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-01-12T00:47:17.556 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-12T00:47:17.846 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:47:17.846 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":163,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T22:48:15.884287+0000","last_up_change":"2024-01-11T21:44:13.948237+0000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T21:44:14.674056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-11T21:44:19.605189+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-01-11T21:44:25.541935+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20057},{"type":"v1","addr":"172.21.15.178:6806","nonce":20057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20057},{"type":"v1","addr":"172.21.15.178:6810","nonce":20057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20057},{"type":"v1","addr":"172.21.15.178:6815","nonce":20057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20057},{"type":"v1","addr":"172.21.15.178:6813","nonce":20057}]},"public_addr":"172.21.15.178:6806/20057","cluster_addr":"172.21.15.178:6810/20057","heartbeat_back_addr":"172.21.15.178:6815/20057","heartbeat_front_addr":"172.21.15.178:6813/20057","state":["exists","up"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20056},{"type":"v1","addr":"172.21.15.178:6801","nonce":20056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20056},{"type":"v1","addr":"172.21.15.178:6803","nonce":20056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":20056},{"type":"v1","addr":"172.21.15.178:6811","nonce":20056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6805","nonce":20056},{"type":"v1","addr":"172.21.15.178:6807","nonce":20056}]},"public_addr":"172.21.15.178:6801/20056","cluster_addr":"172.21.15.178:6803/20056","heartbeat_back_addr":"172.21.15.178:6811/20056","heartbeat_front_addr":"172.21.15.178:6807/20056","state":["exists","up"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20058},{"type":"v1","addr":"172.21.15.178:6817","nonce":20058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20058},{"type":"v1","addr":"172.21.15.178:6819","nonce":20058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20058},{"type":"v1","addr":"172.21.15.178:6823","nonce":20058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20058},{"type":"v1","addr":"172.21.15.178:6821","nonce":20058}]},"public_addr":"172.21.15.178:6817/20058","cluster_addr":"172.21.15.178:6819/20058","heartbeat_back_addr":"172.21.15.178:6823/20058","heartbeat_front_addr":"172.21.15.178:6821/20058","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.562572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.495598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.468222+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":4,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-12T00:47:17.860 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-12T00:47:17.860 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:47:18.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:47:18.147 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:47:18.153 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:18.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:18.156 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5652,"stamp":"2024-01-12T00:47:17.619445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27451,"ondisk_log_size":27451,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"160'1076","reported_seq":2021,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991884+0000","last_change":"2024-01-11T21:49:51.632059+0000","last_active":"2024-01-11T22:48:10.991884+0000","last_peered":"2024-01-11T22:48:10.991884+0000","last_clean":"2024-01-11T22:48:10.991884+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-11T22:48:10.991884+0000","last_undegraded":"2024-01-11T22:48:10.991884+0000","last_fullsized":"2024-01-11T22:48:10.991884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:33:17.886132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"160'1407","reported_seq":2339,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991826+0000","last_change":"2024-01-11T21:49:51.631980+0000","last_active":"2024-01-11T22:48:10.991826+0000","last_peered":"2024-01-11T22:48:10.991826+0000","last_clean":"2024-01-11T22:48:10.991826+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-11T22:48:10.991826+0000","last_undegraded":"2024-01-11T22:48:10.991826+0000","last_fullsized":"2024-01-11T22:48:10.991826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:21.279733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8957,"ondisk_log_size":8957,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740572,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.58799999999999997,"5min":0.65600000000000003,"15min":1.466},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.625,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Jan 12 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.42899999999999999},"max":{"1min":0.78500000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.47899999999999998},"max":{"1min":0.746,"5min":0.90600000000000003,"15min":1.026},"last":0.71499999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740570,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:46:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.41799999999999998,"5min":0.40799999999999997,"15min":0.39200000000000002},"max":{"1min":0.73699999999999999,"5min":0.754,"15min":0.77300000000000002},"last":0.504},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.51900000000000002,"5min":0.51100000000000001,"15min":0.47099999999999997},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":0.81499999999999995},"last":0.626}]},{"osd":2,"last update":"Fri Jan 12 00:46:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.67100000000000004,"5min":0.71399999999999997,"15min":0.97299999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66700000000000004,"15min":0.67100000000000004},"min":{"1min":0.57499999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73799999999999999,"5min":1.046,"15min":1.046},"last":0.71399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359740573,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.47299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.70399999999999996,"5min":0.79200000000000004,"15min":0.83999999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.42799999999999999},"max":{"1min":0.70699999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61799999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.497,"5min":0.441,"15min":0.441},"max":{"1min":0.751,"5min":0.751,"15min":0.78000000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63200000000000001,"15min":0.63700000000000001},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.45300000000000001},"max":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.95199999999999996},"last":0.91600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:47:18.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:47:18.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:18.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:18.442 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:47:18.445 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:47:18.456 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5652,"stamp":"2024-01-12T00:47:17.619445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27451,"ondisk_log_size":27451,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"160'1076","reported_seq":2021,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991884+0000","last_change":"2024-01-11T21:49:51.632059+0000","last_active":"2024-01-11T22:48:10.991884+0000","last_peered":"2024-01-11T22:48:10.991884+0000","last_clean":"2024-01-11T22:48:10.991884+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-11T22:48:10.991884+0000","last_undegraded":"2024-01-11T22:48:10.991884+0000","last_fullsized":"2024-01-11T22:48:10.991884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:33:17.886132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"160'1407","reported_seq":2339,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991826+0000","last_change":"2024-01-11T21:49:51.631980+0000","last_active":"2024-01-11T22:48:10.991826+0000","last_peered":"2024-01-11T22:48:10.991826+0000","last_clean":"2024-01-11T22:48:10.991826+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-11T22:48:10.991826+0000","last_undegraded":"2024-01-11T22:48:10.991826+0000","last_fullsized":"2024-01-11T22:48:10.991826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:21.279733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8957,"ondisk_log_size":8957,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740572,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.58799999999999997,"5min":0.65600000000000003,"15min":1.466},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.625,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Jan 12 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.42899999999999999},"max":{"1min":0.78500000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.47899999999999998},"max":{"1min":0.746,"5min":0.90600000000000003,"15min":1.026},"last":0.71499999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740570,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:46:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.41799999999999998,"5min":0.40799999999999997,"15min":0.39200000000000002},"max":{"1min":0.73699999999999999,"5min":0.754,"15min":0.77300000000000002},"last":0.504},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.51900000000000002,"5min":0.51100000000000001,"15min":0.47099999999999997},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":0.81499999999999995},"last":0.626}]},{"osd":2,"last update":"Fri Jan 12 00:46:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.67100000000000004,"5min":0.71399999999999997,"15min":0.97299999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66700000000000004,"15min":0.67100000000000004},"min":{"1min":0.57499999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73799999999999999,"5min":1.046,"15min":1.046},"last":0.71399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359740573,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.47299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.70399999999999996,"5min":0.79200000000000004,"15min":0.83999999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.42799999999999999},"max":{"1min":0.70699999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61799999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.497,"5min":0.441,"15min":0.441},"max":{"1min":0.751,"5min":0.751,"15min":0.78000000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63200000000000001,"15min":0.63700000000000001},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.45300000000000001},"max":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.95199999999999996},"last":0.91600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:47:18.457 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-12T00:47:18.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:47:18.737 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:47:18.740 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:47:18.750 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5652,"stamp":"2024-01-12T00:47:17.619445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27451,"ondisk_log_size":27451,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"160'1076","reported_seq":2021,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991884+0000","last_change":"2024-01-11T21:49:51.632059+0000","last_active":"2024-01-11T22:48:10.991884+0000","last_peered":"2024-01-11T22:48:10.991884+0000","last_clean":"2024-01-11T22:48:10.991884+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-11T22:48:10.991884+0000","last_undegraded":"2024-01-11T22:48:10.991884+0000","last_fullsized":"2024-01-11T22:48:10.991884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:33:17.886132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"160'1407","reported_seq":2339,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991826+0000","last_change":"2024-01-11T21:49:51.631980+0000","last_active":"2024-01-11T22:48:10.991826+0000","last_peered":"2024-01-11T22:48:10.991826+0000","last_clean":"2024-01-11T22:48:10.991826+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-11T22:48:10.991826+0000","last_undegraded":"2024-01-11T22:48:10.991826+0000","last_fullsized":"2024-01-11T22:48:10.991826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:21.279733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8957,"ondisk_log_size":8957,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740572,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.58799999999999997,"5min":0.65600000000000003,"15min":1.466},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":0.625,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Jan 12 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.42899999999999999},"max":{"1min":0.78500000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.47899999999999998},"max":{"1min":0.746,"5min":0.90600000000000003,"15min":1.026},"last":0.71499999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740570,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:46:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.41799999999999998,"5min":0.40799999999999997,"15min":0.39200000000000002},"max":{"1min":0.73699999999999999,"5min":0.754,"15min":0.77300000000000002},"last":0.504},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.51900000000000002,"5min":0.51100000000000001,"15min":0.47099999999999997},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":0.81499999999999995},"last":0.626}]},{"osd":2,"last update":"Fri Jan 12 00:46:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.67100000000000004,"5min":0.71399999999999997,"15min":0.97299999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66700000000000004,"15min":0.67100000000000004},"min":{"1min":0.57499999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73799999999999999,"5min":1.046,"15min":1.046},"last":0.71399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359740573,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.47299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.70399999999999996,"5min":0.79200000000000004,"15min":0.83999999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.42799999999999999},"max":{"1min":0.70699999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61799999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.497,"5min":0.441,"15min":0.441},"max":{"1min":0.751,"5min":0.751,"15min":0.78000000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63200000000000001,"15min":0.63700000000000001},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.45300000000000001},"max":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.95199999999999996},"last":0.91600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:47:18.751 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-12T00:47:19.030 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:47:19.030 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":163,"fsid":"f4a6b9d5-8d9a-4520-875a-7dbacea84608","created":"2024-01-11T21:44:04.490602+0000","modified":"2024-01-11T22:48:15.884287+0000","last_up_change":"2024-01-11T21:44:13.948237+0000","last_in_change":"2024-01-11T21:44:05.231824+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T21:44:14.674056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-11T21:44:19.605189+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-01-11T21:44:25.541935+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"fe2d4fe9-3870-4d34-b50c-a2823d9a8ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20057},{"type":"v1","addr":"172.21.15.178:6806","nonce":20057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20057},{"type":"v1","addr":"172.21.15.178:6810","nonce":20057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20057},{"type":"v1","addr":"172.21.15.178:6815","nonce":20057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20057},{"type":"v1","addr":"172.21.15.178:6813","nonce":20057}]},"public_addr":"172.21.15.178:6806/20057","cluster_addr":"172.21.15.178:6810/20057","heartbeat_back_addr":"172.21.15.178:6815/20057","heartbeat_front_addr":"172.21.15.178:6813/20057","state":["exists","up"]},{"osd":1,"uuid":"ceb020ed-023b-4e95-8ca5-6b2accf2a58f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20056},{"type":"v1","addr":"172.21.15.178:6801","nonce":20056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20056},{"type":"v1","addr":"172.21.15.178:6803","nonce":20056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":20056},{"type":"v1","addr":"172.21.15.178:6811","nonce":20056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6805","nonce":20056},{"type":"v1","addr":"172.21.15.178:6807","nonce":20056}]},"public_addr":"172.21.15.178:6801/20056","cluster_addr":"172.21.15.178:6803/20056","heartbeat_back_addr":"172.21.15.178:6811/20056","heartbeat_front_addr":"172.21.15.178:6807/20056","state":["exists","up"]},{"osd":2,"uuid":"dc40179d-412f-444c-b8ca-08f7873edc3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20058},{"type":"v1","addr":"172.21.15.178:6817","nonce":20058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20058},{"type":"v1","addr":"172.21.15.178:6819","nonce":20058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20058},{"type":"v1","addr":"172.21.15.178:6823","nonce":20058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20058},{"type":"v1","addr":"172.21.15.178:6821","nonce":20058}]},"public_addr":"172.21.15.178:6817/20058","cluster_addr":"172.21.15.178:6819/20058","heartbeat_back_addr":"172.21.15.178:6823/20058","heartbeat_front_addr":"172.21.15.178:6821/20058","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.562572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.495598+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T21:44:10.468222+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":4,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-12T00:47:19.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:19.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:19.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:19.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:20.044 INFO:tasks.ceph:Scrubbing osd.0 2024-01-12T00:47:20.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-12T00:47:20.131 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:20.131+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:20.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:20.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:21.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:21.143+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:21.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:21.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:22.147 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:22.147+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:22.326 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:22.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:23.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:23.123+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:23.327 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:23.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:24.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:24.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:24.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:24.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:25.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:25.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:25.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:25.315+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:26.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:26.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:26.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:26.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:27.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:27.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:27.293 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:27.291+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:28.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:28.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:28.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:28.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:29.066 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:29.063+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:29.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:29.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:30.113 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:30.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:30.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:30.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:31.140 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:31.139+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:31.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:31.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:32.182 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:32.179+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:32.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:32.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:33.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:33.195+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:33.234 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:33.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:34.158 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:34.155+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:34.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:34.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:35.154 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:35.151+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:35.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:35.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:36.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:36.163+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:36.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:36.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:37.118 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:37.115+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:37.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:37.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:38.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:38.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:38.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:38.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:39.072 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:39.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:39.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:39.247+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:40.092 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:40.091+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:40.268 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:40.267+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:41.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:41.075+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:41.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:41.263+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:42.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:42.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:42.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:42.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:43.081 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:43.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:43.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:43.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:44.111 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:44.111+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:44.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:44.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:45.107 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:45.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:45.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:45.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:46.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:46.087+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:46.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:46.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:47.049 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:47.046+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:47.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:47.234+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:48.098 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:48.094+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:48.198 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:48.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:49.088 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:49.086+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:49.157 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:49.154+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:50.099 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:50.098+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:50.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:50.178+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:51.112 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:51.110+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:51.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:51.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:52.073 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:52.070+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:52.187 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:52.186+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:53.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:53.102+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:53.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:53.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:54.091 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:54.090+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:54.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:54.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:55.055 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:55.054+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:55.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:55.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:56.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:56.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:56.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:56.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:57.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:57.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:57.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:57.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:58.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:58.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:58.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:58.230+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:59.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:47:59.082+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:47:59.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:47:59.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:00.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:00.126+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:00.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:00.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:01.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:01.166+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:01.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:01.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:02.210 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:02.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:02.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:02.238+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:03.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:03.162+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:03.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:03.226+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:04.171 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:04.166+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:04.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:04.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:05.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:05.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:05.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:05.302+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:06.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:06.234+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:06.344 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:06.342+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:07.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:07.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:07.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:07.298+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:08.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:08.266+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:08.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:08.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:09.279 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:09.278+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:09.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:09.318+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:10.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:10.234+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:10.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:10.282+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:11.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:11.222+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:11.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:11.246+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:12.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:12.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:12.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:12.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:13.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:13.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:13.284 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:13.282+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:14.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:14.142+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:14.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:14.326+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:15.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:15.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:15.303 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:15.302+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:16.167 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:16.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:16.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:16.346+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:17.137 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:17.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:17.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:17.350+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:18.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:18.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:18.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:18.338+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:19.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:19.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:19.322 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:19.318+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:20.129 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:20.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:20.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:20.286+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:21.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:21.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:21.254 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:21.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:22.125 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:22.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:22.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:22.290+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:23.111 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:23.110+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:23.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:23.302+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:24.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:24.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:24.269 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:24.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:25.032 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:25.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:25.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:25.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:26.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:26.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:26.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:26.330+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:27.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:27.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:27.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:27.282+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:28.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:28.038+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:28.305 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:28.302+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:29.075 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:29.074+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:29.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:29.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:30.040 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:30.038+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:30.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:30.294+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:31.088 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:31.086+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:31.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:31.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:32.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:32.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:32.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:32.262+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:33.104 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:33.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:33.251 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:33.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:34.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:34.078+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:34.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:34.262+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:35.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:35.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:35.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:35.310+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:36.121 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:36.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:36.279 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:36.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:37.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:37.094+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:37.325 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:37.322+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:38.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:38.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:38.307 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:38.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:39.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:39.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:39.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:39.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:40.206 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:40.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:40.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:40.277+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:41.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:41.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:41.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:41.241+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:42.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:42.217+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:42.296 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:42.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:43.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:43.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:43.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:43.305+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:44.190 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:44.185+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:44.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:44.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:45.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:45.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:45.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:45.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:46.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:46.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:46.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:46.353+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:47.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:47.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:47.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:47.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:48.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:48.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:48.385 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:48.381+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:49.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:49.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:49.391 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:49.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:50.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:50.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:50.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:50.421+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:51.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:51.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:51.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:51.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:52.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:52.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:52.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:52.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:53.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:53.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:53.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:53.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:54.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:54.393+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:54.407 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:54.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:55.377 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:55.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:55.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:55.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:56.414 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:56.409+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:56.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:56.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:57.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:57.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:57.422 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:57.417+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:58.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:58.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:58.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:58.393+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:59.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:48:59.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:48:59.370 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:48:59.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:00.342 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:00.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:00.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:00.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:01.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:01.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:01.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:01.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:02.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:02.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:02.460 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:02.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:03.248 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:03.245+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:03.474 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:03.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:04.224 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:04.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:04.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:04.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:05.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:05.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:05.446 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:05.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:06.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:06.269+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:06.431 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:06.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:07.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:07.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:07.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:07.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:08.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:08.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:08.364 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:08.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:09.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:09.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:09.361 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:09.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:10.270 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:10.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:10.342 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:10.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:11.276 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:11.273+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:11.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:11.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:12.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:12.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:12.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:12.353+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:13.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:13.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:13.357 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:13.353+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:14.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:14.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:14.365 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:14.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:15.291 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:15.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:15.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:15.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:16.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:16.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:16.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:16.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:17.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:17.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:17.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:17.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:18.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:18.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:18.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:18.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:19.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:19.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:19.435 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:19.433+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-12T00:49:20.070 INFO:tasks.ceph:Scrubbing osd.1 2024-01-12T00:49:20.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-12T00:49:20.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:20.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:20.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:20.465+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:21.233 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:21.229+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:21.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:21.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:22.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:22.213+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:22.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:22.409+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:23.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:23.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:23.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:23.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:24.188 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:24.185+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:24.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:24.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:25.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:25.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:25.486 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:25.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:26.174 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:26.169+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:26.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:26.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:27.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:27.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:27.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:27.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:28.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:28.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:28.477 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:28.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:29.178 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:29.173+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:29.493 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:29.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:30.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:30.177+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:30.505 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:30.501+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:31.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:31.193+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:31.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:31.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:32.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:32.212+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:32.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:32.444+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:33.265 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:33.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:33.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:33.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:34.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:34.216+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:34.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:34.404+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:35.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:35.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:35.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:35.384+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:36.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:36.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:36.383 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:36.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:37.249 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:37.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:37.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:37.340+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:38.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:38.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:38.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:38.364+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:39.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:39.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:39.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:39.348+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:40.323 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:40.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:40.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:40.348+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:41.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:41.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:41.326 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:41.320+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:42.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:42.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:42.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:42.276+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:43.227 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:43.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:43.301 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:43.296+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:44.230 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:44.224+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:44.297 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:44.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:45.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:45.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:45.286 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:45.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:46.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:46.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:46.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:46.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:47.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:47.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:47.273 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:47.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:48.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:48.228+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:48.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:48.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:49.230 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:49.224+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:49.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:49.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:50.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:50.232+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:50.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:50.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:51.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:51.200+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:51.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:51.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:52.191 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:52.184+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:52.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:52.332+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:53.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:53.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:53.290 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:53.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:54.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:54.140+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:54.256 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:54.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:55.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:55.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:55.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:55.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:56.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:56.216+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:56.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:56.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:57.211 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:57.208+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:57.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:57.284+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:58.220 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:58.216+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:58.249 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:58.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:59.215 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:49:59.208+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:49:59.252 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:49:59.248+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:00.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:00.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:00.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:00.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:01.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:01.220+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:01.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:01.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:02.260 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:02.256+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:02.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:02.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:03.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:03.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:03.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:03.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:04.238 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:04.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:04.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:04.264+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:05.243 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:05.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:05.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:05.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:06.224 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:06.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:06.257 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:06.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:07.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:07.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:07.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:07.280+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:08.155 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:08.152+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:08.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:08.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:09.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:09.160+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:09.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:09.292+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:10.199 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:10.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:10.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:10.304+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:11.194 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:11.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:11.299 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:11.300+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:12.186 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:12.184+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:12.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:12.328+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:13.183 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:13.180+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:13.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:13.308+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:14.167 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:14.168+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:14.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:14.316+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:15.127 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:15.128+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:15.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:15.312+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:16.124 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:16.124+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:16.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:16.284+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:17.122 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:17.120+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:17.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:17.324+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:18.093 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:18.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:18.364 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:18.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:19.062 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:19.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:19.365 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:19.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:20.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:20.084+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:20.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:20.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:21.093 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:21.092+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:21.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:21.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:22.053 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:22.052+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:22.382 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:22.380+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:23.087 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:23.088+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:23.409 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:23.408+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:24.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:24.060+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:24.435 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:24.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:25.022 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:25.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:25.440 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:25.439+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:26.068 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:26.067+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:26.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:26.491+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:27.061 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:27.059+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:27.449 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:27.447+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:28.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:28.055+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:28.455 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:28.455+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:29.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:29.007+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:29.500 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:29.499+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:29.999 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:29.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:30.483 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:30.483+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:30.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:30.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:31.509 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:31.507+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:31.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:31.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:32.505 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:32.503+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:32.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:32.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:33.519 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:33.519+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:33.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:33.963+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:34.565 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:34.563+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:34.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:34.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:35.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:35.615+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:35.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:35.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:36.664 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:36.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:36.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:36.955+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:37.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:37.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:37.987 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:37.983+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:38.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:38.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:38.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:38.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:39.788 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:39.787+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:39.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:39.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:40.780 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:40.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:41.020 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:41.019+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:41.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:41.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:41.987 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:41.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:42.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:42.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:42.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:42.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:43.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:43.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:44.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:44.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:44.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:44.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:44.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:44.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:45.745 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:45.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:46.004 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:46.003+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:46.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:46.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:46.996 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:46.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:47.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:47.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:48.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:48.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:48.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:48.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:48.967 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:48.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:49.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:49.779+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:50.013 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:50.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:50.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:50.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:51.011 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:51.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:51.718 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:51.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:52.026 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:52.023+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:52.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:52.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:53.076 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:53.075+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:53.697 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:53.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:54.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:54.083+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:54.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:54.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:55.109 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:55.107+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:55.725 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:55.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:56.105 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:56.103+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:56.738 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:56.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:57.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:57.079+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:57.719 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:57.719+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:58.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:58.043+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:58.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:58.759+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:59.056 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:50:59.055+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:50:59.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:50:59.771+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:00.085 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:00.083+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:00.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:00.771+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:01.074 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:01.071+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:01.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:01.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:02.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:02.039+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:02.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:02.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:02.997 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:02.995+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:03.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:03.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:03.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:03.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:04.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:04.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:04.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:04.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:05.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:05.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:05.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:05.951+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:06.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:06.659+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:06.927 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:06.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:07.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:07.615+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:07.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:07.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:08.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:08.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:08.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:08.963+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:09.645 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:09.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:09.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:09.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:10.662 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:10.659+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:10.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:10.907+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:11.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:11.683+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:11.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:11.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:12.687 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:12.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:12.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:12.919+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:13.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:13.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:13.923 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:13.923+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:14.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:14.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:14.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:14.947+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:15.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:15.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:15.972 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:15.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:16.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:16.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:16.959 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:16.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:17.671 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:17.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:17.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:17.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:18.652 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:18.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:19.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:19.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:19.683 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:19.682+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:20.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:20.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-12T00:51:20.094 INFO:tasks.ceph:Scrubbing osd.2 2024-01-12T00:51:20.095 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-12T00:51:20.242 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-01-12T00:51:20.242 INFO:teuthology.orchestra.run.smithi178.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-12T00:51:20.242 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-01-12T00:51:20.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-12T00:51:20.564 INFO:teuthology.orchestra.run.smithi178.stderr:instructed osd(s) 2 to deep-scrub 2024-01-12T00:51:20.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:51:20.654 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:20.650+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:20.881 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:51:20.884 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:51:20.895 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5773,"stamp":"2024-01-12T00:51:19.687086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27451,"ondisk_log_size":27451,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003288"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"160'1076","reported_seq":2021,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991884+0000","last_change":"2024-01-11T21:49:51.632059+0000","last_active":"2024-01-11T22:48:10.991884+0000","last_peered":"2024-01-11T22:48:10.991884+0000","last_clean":"2024-01-11T22:48:10.991884+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-11T22:48:10.991884+0000","last_undegraded":"2024-01-11T22:48:10.991884+0000","last_fullsized":"2024-01-11T22:48:10.991884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:33:17.886132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"160'1407","reported_seq":2339,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.991826+0000","last_change":"2024-01-11T21:49:51.631980+0000","last_active":"2024-01-11T22:48:10.991826+0000","last_peered":"2024-01-11T22:48:10.991826+0000","last_clean":"2024-01-11T22:48:10.991826+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-11T22:48:10.991826+0000","last_undegraded":"2024-01-11T22:48:10.991826+0000","last_fullsized":"2024-01-11T22:48:10.991826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:21.279733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8957,"ondisk_log_size":8957,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740620,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.65500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.44},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.84299999999999997},"last":0.58799999999999997}]},{"osd":1,"last update":"Fri Jan 12 00:50:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.45300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.81999999999999995},"last":0.629},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.80500000000000005,"5min":1.008,"15min":1.026},"last":0.73299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740618,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.439,"5min":0.41299999999999998,"15min":0.39200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.62},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.53100000000000003,"5min":0.5,"15min":0.48999999999999999},"max":{"1min":0.67900000000000005,"5min":0.77600000000000002,"15min":0.81100000000000005},"last":0.77800000000000002}]},{"osd":2,"last update":"Fri Jan 12 00:50:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.45700000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66700000000000004,"5min":0.73499999999999999,"15min":0.97299999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.58499999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.754,"5min":0.86299999999999999,"15min":1.046},"last":0.70599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359740621,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.46200000000000002,"5min":0.433,"15min":0.42399999999999999},"max":{"1min":0.58599999999999997,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.69799999999999995,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.67300000000000004}]},{"osd":2,"last update":"Fri Jan 12 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.441},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.45700000000000002},"max":{"1min":0.71899999999999997,"5min":0.91600000000000004,"15min":0.95199999999999996},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:51:20.899 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.900 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.900 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.900 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.900 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.900 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.901 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.901 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.901 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.901 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.901 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.902 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.902 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:20.902 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:51:21.086 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:21.082+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:21.650 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:21.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:22.058 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:22.054+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:22.649 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:22.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:23.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:23.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:23.642 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:23.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:23.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:23.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:24.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:24.614+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:25.006 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:25.002+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:25.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:25.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:26.037 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:26.034+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:26.641 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:26.638+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:26.998 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:26.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:27.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:27.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:27.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:27.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:28.689 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:28.686+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:28.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:28.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:29.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:29.666+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:29.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:29.934+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:30.701 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:30.698+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:30.945 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:30.942+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:31.707 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:31.702+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:31.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:31.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:32.665 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:32.662+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:32.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:32.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:33.711 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:33.710+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:33.997 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:33.994+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:34.682 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:34.678+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:34.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:34.958+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:35.691 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:35.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:36.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:35.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:36.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:36.642+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:36.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:36.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:37.608 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:37.606+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:37.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:37.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:38.648 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:38.646+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:38.905 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:38.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:39.644 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:39.642+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:39.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:39.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:40.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:40.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:40.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:40.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:40.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:51:41.199 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:51:41.202 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:51:41.213 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5783,"stamp":"2024-01-12T00:51:39.692692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27524,"ondisk_log_size":27524,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003231"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9030,"ondisk_log_size":9030,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740624,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.67700000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.44},"max":{"1min":0.67200000000000004,"5min":0.70999999999999996,"15min":0.84299999999999997},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Jan 12 00:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71999999999999997,"5min":0.79400000000000004,"15min":0.81999999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69699999999999995},"min":{"1min":0.624,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.79200000000000004,"5min":1.008,"15min":1.026},"last":0.628}]}]},{"osd":1,"up_from":8,"seq":34359740622,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.439,"5min":0.41299999999999998,"15min":0.39200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.53100000000000003,"5min":0.5,"15min":0.48999999999999999},"max":{"1min":0.67900000000000005,"5min":0.77600000000000002,"15min":0.81100000000000005},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Jan 12 00:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.51400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71099999999999997,"5min":0.73499999999999999,"15min":0.97299999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.58899999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86299999999999999,"15min":1.046},"last":0.64900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359740625,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.46200000000000002,"5min":0.433,"15min":0.42399999999999999},"max":{"1min":0.58599999999999997,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.69799999999999995,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.47999999999999998}]},{"osd":2,"last update":"Fri Jan 12 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.441},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.45700000000000002},"max":{"1min":0.71899999999999997,"5min":0.91600000000000004,"15min":0.95199999999999996},"last":0.73299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:51:41.214 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.214 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.214 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.215 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.215 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.215 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.215 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.215 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.216 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.216 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.216 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:51:41.216 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:51:41.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:41.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:41.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:41.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:42.618 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:42.614+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:42.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:42.806+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:43.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:43.610+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:43.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:43.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:44.574 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:44.570+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:44.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:44.814+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:45.547 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:45.546+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:45.796 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:45.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:46.586 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:46.582+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:46.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:46.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:47.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:47.626+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:47.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:47.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:48.657 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:48.654+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:48.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:48.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:49.674 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:49.670+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:49.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:49.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:50.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:50.658+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:50.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:50.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:51.694 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:51.690+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:51.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:51.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:52.681 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:52.678+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:52.880 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:52.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:53.709 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:53.706+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:53.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:53.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:54.730 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:54.726+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:54.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:54.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:55.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:55.714+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:55.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:55.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:56.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:56.738+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:56.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:56.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:57.756 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:57.754+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:57.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:57.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:58.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:58.762+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:58.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:58.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:59.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:51:59.734+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:51:59.950 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:51:59.946+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:00.771 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:00.770+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:00.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:00.906+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:01.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:52:01.511 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:52:01.514 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:52:01.525 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5793,"stamp":"2024-01-12T00:51:59.698271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27524,"ondisk_log_size":27524,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9030,"ondisk_log_size":9030,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740628,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.67700000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.44},"max":{"1min":0.67200000000000004,"5min":0.70999999999999996,"15min":0.84299999999999997},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Jan 12 00:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71999999999999997,"5min":0.79400000000000004,"15min":0.81999999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69699999999999995},"min":{"1min":0.624,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.79200000000000004,"5min":1.008,"15min":1.026},"last":0.67500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740626,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51800000000000002,"15min":0.52800000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.39200000000000002},"max":{"1min":0.62,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.624},"min":{"1min":0.502,"5min":0.5,"15min":0.48999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.81100000000000005},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jan 12 00:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.51400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71099999999999997,"5min":0.73499999999999999,"15min":0.97299999999999998},"last":0.501},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.58899999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86299999999999999,"15min":1.046},"last":0.63}]}]},{"osd":0,"up_from":8,"seq":34359740629,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.46200000000000002,"5min":0.433,"15min":0.42399999999999999},"max":{"1min":0.58599999999999997,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.69799999999999995,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.623}]},{"osd":2,"last update":"Fri Jan 12 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.441},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.45700000000000002},"max":{"1min":0.71899999999999997,"5min":0.91600000000000004,"15min":0.95199999999999996},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:52:01.527 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.527 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.527 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.527 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.527 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.527 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.528 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.528 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.528 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.528 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.528 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:01.529 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:52:01.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:01.810+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:01.894 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:01.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:02.841 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:02.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:02.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:02.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:03.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:03.854+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:03.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:03.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:04.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:04.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:04.892 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:04.890+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:05.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:05.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:05.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:05.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:06.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:06.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:06.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:06.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:07.863 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:07.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:07.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:07.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:08.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:08.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:08.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:08.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:09.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:09.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:09.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:09.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:10.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:10.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:10.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:10.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:11.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:11.833+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:11.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:11.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:12.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:12.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:12.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:12.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:13.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:13.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:13.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:13.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:14.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:14.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:14.931 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:14.929+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:15.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:15.749+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:15.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:15.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:16.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:16.749+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:16.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:16.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:17.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:17.749+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:17.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:17.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:18.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:18.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:18.892 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:18.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:19.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:19.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:19.850 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:19.845+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:20.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:20.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:20.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:20.821+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:21.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:52:21.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:21.749+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:21.794 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:21.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:21.831 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:52:21.834 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:52:21.845 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5804,"stamp":"2024-01-12T00:52:21.704152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27524,"ondisk_log_size":27524,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003170"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9030,"ondisk_log_size":9030,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740633,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.67700000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.44},"max":{"1min":0.67200000000000004,"5min":0.70999999999999996,"15min":0.84299999999999997},"last":0.625}]},{"osd":1,"last update":"Fri Jan 12 00:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71999999999999997,"5min":0.79400000000000004,"15min":0.81999999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69699999999999995},"min":{"1min":0.624,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.79200000000000004,"5min":1.008,"15min":1.026},"last":0.74399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359740631,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51800000000000002,"15min":0.52800000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.39200000000000002},"max":{"1min":0.62,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.503},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.624},"min":{"1min":0.502,"5min":0.5,"15min":0.48999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.81100000000000005},"last":0.63600000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.51400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71099999999999997,"5min":0.73499999999999999,"15min":0.97299999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.58899999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86299999999999999,"15min":1.046},"last":0.59099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359740634,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53800000000000003,"15min":0.54500000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.69999999999999996,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.67300000000000004,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Jan 12 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.68600000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63700000000000001},"min":{"1min":0.56799999999999995,"5min":0.48899999999999999,"15min":0.45700000000000002},"max":{"1min":0.73299999999999998,"5min":0.91600000000000004,"15min":0.95199999999999996},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:52:21.846 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.846 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.847 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.847 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.847 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.847 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.847 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.847 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.848 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.848 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.848 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:21.848 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:52:22.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:22.713+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:22.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:22.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:23.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:23.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:23.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:23.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:24.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:24.737+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:24.783 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:24.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:25.776 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:25.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:25.831 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:25.829+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:26.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:26.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:26.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:26.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:27.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:27.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:27.856 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:27.853+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:28.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:28.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:28.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:28.849+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:29.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:29.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:29.835 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:29.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:30.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:30.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:30.809 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:30.805+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:31.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:31.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:31.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:31.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:32.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:32.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:32.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:32.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:33.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:33.789+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:33.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:33.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:34.777 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:34.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:34.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:34.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:35.818 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:35.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:35.908 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:35.905+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:36.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:36.845+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:36.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:36.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:37.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:37.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:37.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:37.881+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:38.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:38.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:38.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:38.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:39.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:39.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:39.845 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:39.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:40.823 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:40.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:40.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:40.861+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:41.790 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:41.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:41.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:41.817+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:41.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:52:42.142 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:52:42.145 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:52:42.155 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5814,"stamp":"2024-01-12T00:52:41.709824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27524,"ondisk_log_size":27524,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003244"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9030,"ondisk_log_size":9030,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740637,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67600000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.84299999999999997},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri Jan 12 00:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.68999999999999995,"5min":0.79400000000000004,"15min":0.81999999999999995},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.56000000000000005,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.76000000000000001,"5min":1.008,"15min":1.008},"last":0.66500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740635,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51800000000000002,"15min":0.52800000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.39200000000000002},"max":{"1min":0.62,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.624},"min":{"1min":0.502,"5min":0.5,"15min":0.48999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.81100000000000005},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Jan 12 00:52:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.5,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.72099999999999997,"5min":0.73499999999999999,"15min":0.97299999999999998},"last":0.498},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67200000000000004},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.46300000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":1.046},"last":0.629}]}]},{"osd":0,"up_from":8,"seq":34359740638,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53800000000000003,"15min":0.54500000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.69999999999999996,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.67300000000000004,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.47499999999999998}]},{"osd":2,"last update":"Fri Jan 12 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.68600000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63700000000000001},"min":{"1min":0.56799999999999995,"5min":0.48899999999999999,"15min":0.45700000000000002},"max":{"1min":0.73299999999999998,"5min":0.91600000000000004,"15min":0.95199999999999996},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:52:42.157 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.157 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.157 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.157 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.158 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.158 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.158 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.158 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.158 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.159 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.159 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:52:42.159 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:52:42.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:42.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:42.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:42.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:43.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:43.785+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:43.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:43.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:44.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:44.765+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:44.874 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:44.869+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:45.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:45.745+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:45.898 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:45.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:46.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:46.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:46.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:46.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:47.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:47.773+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:47.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:47.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:48.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:48.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:48.801 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:48.797+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:49.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:49.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:49.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:49.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:50.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:50.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:50.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:50.753+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:51.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:51.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:51.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:51.753+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:52.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:52.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:52.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:52.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:53.710 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:53.705+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:53.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:53.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:54.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:54.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:54.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:54.741+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:55.665 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:55.661+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:55.724 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:55.721+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:56.695 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:56.689+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:56.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:56.761+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:57.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:57.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:57.815 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:57.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:58.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:58.701+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:58.865 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:58.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:59.687 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:52:59.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:52:59.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:52:59.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:00.670 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:00.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:00.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:00.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:01.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:01.677+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:01.890 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:01.885+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:02.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:53:02.458 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:53:02.461 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:53:02.471 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5824,"stamp":"2024-01-12T00:53:01.715178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27524,"ondisk_log_size":27524,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003149"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9030,"ondisk_log_size":9030,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740641,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67600000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.84299999999999997},"last":0.59999999999999998}]},{"osd":1,"last update":"Fri Jan 12 00:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.68999999999999995,"5min":0.79400000000000004,"15min":0.81999999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.56000000000000005,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.76000000000000001,"5min":1.008,"15min":1.008},"last":0.71399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740639,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.39200000000000002},"max":{"1min":0.63900000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61499999999999999,"15min":0.624},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.5},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.81100000000000005},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Jan 12 00:52:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.5,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.72099999999999997,"5min":0.73499999999999999,"15min":0.97299999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67200000000000004},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.46300000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":1.046},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359740642,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53800000000000003,"15min":0.54500000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.69999999999999996,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.67300000000000004,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.58999999999999997}]},{"osd":2,"last update":"Fri Jan 12 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.68600000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.752},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63700000000000001},"min":{"1min":0.56799999999999995,"5min":0.48899999999999999,"15min":0.45700000000000002},"max":{"1min":0.73299999999999998,"5min":0.91600000000000004,"15min":0.95199999999999996},"last":0.65100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:53:02.473 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.473 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.473 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.473 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.473 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.474 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.474 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.474 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.474 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.474 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.474 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:02.475 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:53:02.687 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:02.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:02.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:02.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:03.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:03.652+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:03.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:03.840+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:04.636 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:04.632+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:04.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:04.884+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:05.627 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:05.624+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:05.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:05.892+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:06.608 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:06.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:06.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:06.936+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:07.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:07.632+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:07.916 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:07.912+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:08.684 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:08.680+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:08.911 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:08.908+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:09.655 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:09.652+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:09.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:09.884+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:10.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:10.604+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:10.920 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:10.916+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:11.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:11.596+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:11.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:11.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:12.647 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:12.644+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:12.968 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:12.964+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:13.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:13.636+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:14.015 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:14.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:14.634 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:14.628+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:15.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:15.052+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:15.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:15.680+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:16.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:16.100+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:16.689 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:16.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:17.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:17.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:17.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:17.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:18.132 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:18.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:18.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:18.652+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:19.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:19.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:19.657 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:19.652+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:20.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:20.104+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:20.674 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:20.668+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:21.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:21.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:21.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:21.640+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:22.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:22.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:22.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:53:22.678 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:22.672+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:22.767 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:53:22.772 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:53:22.785 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5834,"stamp":"2024-01-12T00:53:21.720795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27524,"ondisk_log_size":27524,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003374"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9030,"ondisk_log_size":9030,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740645,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67600000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.84299999999999997},"last":0.59899999999999998}]},{"osd":1,"last update":"Fri Jan 12 00:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.68999999999999995,"5min":0.79400000000000004,"15min":0.81999999999999995},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.56000000000000005,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.76000000000000001,"5min":1.008,"15min":1.008},"last":0.81699999999999995}]}]},{"osd":1,"up_from":8,"seq":34359740643,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.39200000000000002},"max":{"1min":0.63900000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61499999999999999,"15min":0.624},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.5},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.81100000000000005},"last":0.626}]},{"osd":2,"last update":"Fri Jan 12 00:52:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.5,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.72099999999999997,"5min":0.73499999999999999,"15min":0.97299999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67200000000000004},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.46300000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":1.046},"last":0.73799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740646,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68100000000000005,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.5},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.47499999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.69099999999999995,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.63400000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.53800000000000003,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.64000000000000001},"min":{"1min":0.56999999999999995,"5min":0.48899999999999999,"15min":0.47499999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:53:22.786 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.786 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.787 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.787 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.787 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.787 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.787 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.788 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.788 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.788 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.788 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:22.788 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:53:23.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:23.020+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:23.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:23.696+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:24.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:24.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:24.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:24.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:25.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:25.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:25.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:25.684+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:25.967 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:25.964+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:26.642 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:26.636+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:27.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:27.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:27.693 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:27.688+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:28.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:28.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:28.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:28.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:28.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:28.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:29.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:29.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:30.016 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:30.012+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:30.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:30.660+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:31.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:31.020+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:31.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:31.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:31.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:31.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:32.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:32.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:33.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:33.000+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:33.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:33.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:34.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:34.020+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:34.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:34.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:35.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:35.068+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:35.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:35.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:36.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:36.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:36.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:36.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:37.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:37.088+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:37.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:37.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:38.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:38.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:38.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:38.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:39.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:39.164+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:39.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:39.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:40.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:40.212+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:40.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:40.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:41.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:41.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:41.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:41.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:42.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:42.204+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:42.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:42.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:42.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:53:43.070 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:53:43.073 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:53:43.083 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5844,"stamp":"2024-01-12T00:53:41.726241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27524,"ondisk_log_size":27524,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1974,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.891174+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:10.891174+0000","last_peered":"2024-01-11T22:48:10.891174+0000","last_clean":"2024-01-11T22:48:10.891174+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:10.891174+0000","last_undegraded":"2024-01-11T22:48:10.891174+0000","last_fullsized":"2024-01-11T22:48:10.891174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T00:25:08.496349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1707,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890851+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:10.890851+0000","last_peered":"2024-01-11T22:48:10.890851+0000","last_clean":"2024-01-11T22:48:10.890851+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:10.890851+0000","last_undegraded":"2024-01-11T22:48:10.890851+0000","last_fullsized":"2024-01-11T22:48:10.890851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T06:57:44.996932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1919,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890649+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:10.890649+0000","last_peered":"2024-01-11T22:48:10.890649+0000","last_clean":"2024-01-11T22:48:10.890649+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:10.890649+0000","last_undegraded":"2024-01-11T22:48:10.890649+0000","last_fullsized":"2024-01-11T22:48:10.890649+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T04:16:56.967485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'893","reported_seq":1823,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890409+0000","last_change":"2024-01-11T21:49:51.638231+0000","last_active":"2024-01-11T22:48:10.890409+0000","last_peered":"2024-01-11T22:48:10.890409+0000","last_clean":"2024-01-11T22:48:10.890409+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-11T22:48:10.890409+0000","last_undegraded":"2024-01-11T22:48:10.890409+0000","last_fullsized":"2024-01-11T22:48:10.890409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:05:34.266532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12865,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889533+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:10.889533+0000","last_peered":"2024-01-11T22:48:10.889533+0000","last_clean":"2024-01-11T22:48:10.889533+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:10.889533+0000","last_undegraded":"2024-01-11T22:48:10.889533+0000","last_fullsized":"2024-01-11T22:48:10.889533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:36:55.111249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889326+0000","last_change":"2024-01-11T21:49:51.647778+0000","last_active":"2024-01-11T22:48:10.889326+0000","last_peered":"2024-01-11T22:48:10.889326+0000","last_clean":"2024-01-11T22:48:10.889326+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-11T22:48:10.889326+0000","last_undegraded":"2024-01-11T22:48:10.889326+0000","last_fullsized":"2024-01-11T22:48:10.889326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:19:44.674868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12349,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889817+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:10.889817+0000","last_peered":"2024-01-11T22:48:10.889817+0000","last_clean":"2024-01-11T22:48:10.889817+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:10.889817+0000","last_undegraded":"2024-01-11T22:48:10.889817+0000","last_fullsized":"2024-01-11T22:48:10.889817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:54:46.008602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889200+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:10.889200+0000","last_peered":"2024-01-11T22:48:10.889200+0000","last_clean":"2024-01-11T22:48:10.889200+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:10.889200+0000","last_undegraded":"2024-01-11T22:48:10.889200+0000","last_fullsized":"2024-01-11T22:48:10.889200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T03:08:32.773948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11777,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889515+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:10.889515+0000","last_peered":"2024-01-11T22:48:10.889515+0000","last_clean":"2024-01-11T22:48:10.889515+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:10.889515+0000","last_undegraded":"2024-01-11T22:48:10.889515+0000","last_fullsized":"2024-01-11T22:48:10.889515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T07:08:52.180958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2515,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.889517+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:10.889517+0000","last_peered":"2024-01-11T22:48:10.889517+0000","last_clean":"2024-01-11T22:48:10.889517+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:10.889517+0000","last_undegraded":"2024-01-11T22:48:10.889517+0000","last_fullsized":"2024-01-11T22:48:10.889517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T23:37:22.610162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":282,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-01-11T22:48:10.890104+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:10.890104+0000","last_peered":"2024-01-11T22:48:10.890104+0000","last_clean":"2024-01-11T22:48:10.890104+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:10.890104+0000","last_undegraded":"2024-01-11T22:48:10.890104+0000","last_fullsized":"2024-01-11T22:48:10.890104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T01:49:20.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":592,"num_omap_keys":65,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9030,"ondisk_log_size":9030,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740649,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.443,"5min":0.37,"15min":0.37},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.46400000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58799999999999997}]},{"osd":1,"last update":"Fri Jan 12 00:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.45700000000000002,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.60799999999999998,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":1.008},"last":0.67200000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740647,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.39200000000000002},"max":{"1min":0.63900000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.433},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61499999999999999,"15min":0.624},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.5},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.81100000000000005},"last":0.5}]},{"osd":2,"last update":"Fri Jan 12 00:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.47999999999999998,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76100000000000001,"5min":0.90800000000000003,"15min":1.046},"last":0.55500000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740650,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68100000000000005,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.47499999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.69099999999999995,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.57799999999999996}]},{"osd":2,"last update":"Fri Jan 12 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.53800000000000003,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.64000000000000001},"min":{"1min":0.56999999999999995,"5min":0.48899999999999999,"15min":0.47499999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:53:43.085 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.085 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.085 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.085 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.085 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.086 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.086 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.086 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.086 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.086 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.086 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:53:43.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2024-01-12T00:53:43.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:43.252+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:43.369 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2024-01-12T00:53:43.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2024-01-12T00:53:43.665 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2024-01-12T00:53:43.680 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2024-01-12T00:53:43.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:43.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:43.964 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2024-01-12T00:53:43.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2024-01-12T00:53:44.252 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2024-01-12T00:53:44.262 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2024-01-12T00:53:44.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:44.288+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:44.546 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2024-01-12T00:53:44.556 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2024-01-12T00:53:44.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:44.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:44.837 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2024-01-12T00:53:44.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2024-01-12T00:53:45.134 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-01-12T00:53:45.144 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2024-01-12T00:53:45.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:45.260+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:45.434 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2024-01-12T00:53:45.444 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2024-01-12T00:53:45.722 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2024-01-12T00:53:45.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2024-01-12T00:53:45.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:45.844+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:46.019 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2024-01-12T00:53:46.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2024-01-12T00:53:46.246 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:46.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:46.314 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-01-12T00:53:46.324 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:53:46.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:46.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:47.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:47.240+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:47.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:47.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:48.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:48.196+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:48.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:48.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:49.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:49.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:49.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:49.856+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:50.274 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:50.272+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:50.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:50.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:51.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:51.236+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:51.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:51.876+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:52.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:52.224+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:52.906 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:52.904+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:53.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:53.244+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:53.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:53.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:54.267 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:54.268+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:54.908 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:54.908+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:55.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:55.223+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:55.946 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:55.943+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:56.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:56.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:56.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:56.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:57.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:57.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:57.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:57.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:58.253 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:58.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:58.976 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:58.975+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:59.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:53:59.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:53:59.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:53:59.991+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:00.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:00.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:00.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:00.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:01.281 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:01.279+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:01.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:01.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:02.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:02.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:02.961 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:02.959+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:03.360 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:03.359+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:03.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:03.979+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:04.395 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:04.395+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:05.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:05.011+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:05.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:05.347+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:05.968 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:05.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:06.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:06.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:06.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:54:06.608 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:54:06.611 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:54:06.621 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5856,"stamp":"2024-01-12T00:54:05.732513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003136"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1975,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891845+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:15.891845+0000","last_peered":"2024-01-11T22:48:15.891845+0000","last_clean":"2024-01-11T22:48:15.891845+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:15.891845+0000","last_undegraded":"2024-01-11T22:48:15.891845+0000","last_fullsized":"2024-01-11T22:48:15.891845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1708,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891554+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:15.891554+0000","last_peered":"2024-01-11T22:48:15.891554+0000","last_clean":"2024-01-11T22:48:15.891554+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:15.891554+0000","last_undegraded":"2024-01-11T22:48:15.891554+0000","last_fullsized":"2024-01-11T22:48:15.891554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1920,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891449+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:15.891449+0000","last_peered":"2024-01-11T22:48:15.891449+0000","last_clean":"2024-01-11T22:48:15.891449+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:15.891449+0000","last_undegraded":"2024-01-11T22:48:15.891449+0000","last_fullsized":"2024-01-11T22:48:15.891449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12866,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890524+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:15.890524+0000","last_peered":"2024-01-11T22:48:15.890524+0000","last_clean":"2024-01-11T22:48:15.890524+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:15.890524+0000","last_undegraded":"2024-01-11T22:48:15.890524+0000","last_fullsized":"2024-01-11T22:48:15.890524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2139,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815190+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815190+0000","last_peered":"2024-01-12T00:53:48.815190+0000","last_clean":"2024-01-12T00:53:48.815190+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815190+0000","last_undegraded":"2024-01-12T00:53:48.815190+0000","last_fullsized":"2024-01-12T00:53:48.815190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12350,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891077+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:15.891077+0000","last_peered":"2024-01-11T22:48:15.891077+0000","last_clean":"2024-01-11T22:48:15.891077+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:15.891077+0000","last_undegraded":"2024-01-11T22:48:15.891077+0000","last_fullsized":"2024-01-11T22:48:15.891077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7501,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890630+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:15.890630+0000","last_peered":"2024-01-11T22:48:15.890630+0000","last_clean":"2024-01-11T22:48:15.890630+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:15.890630+0000","last_undegraded":"2024-01-11T22:48:15.890630+0000","last_fullsized":"2024-01-11T22:48:15.890630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11778,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891091+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:15.891091+0000","last_peered":"2024-01-11T22:48:15.891091+0000","last_clean":"2024-01-11T22:48:15.891091+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:15.891091+0000","last_undegraded":"2024-01-11T22:48:15.891091+0000","last_fullsized":"2024-01-11T22:48:15.891091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891410+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:15.891410+0000","last_peered":"2024-01-11T22:48:15.891410+0000","last_clean":"2024-01-11T22:48:15.891410+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:15.891410+0000","last_undegraded":"2024-01-11T22:48:15.891410+0000","last_fullsized":"2024-01-11T22:48:15.891410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891107+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:15.891107+0000","last_peered":"2024-01-11T22:48:15.891107+0000","last_clean":"2024-01-11T22:48:15.891107+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:15.891107+0000","last_undegraded":"2024-01-11T22:48:15.891107+0000","last_fullsized":"2024-01-11T22:48:15.891107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740654,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.443,"5min":0.37,"15min":0.37},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.46400000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.56799999999999995}]},{"osd":1,"last update":"Fri Jan 12 00:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.45700000000000002,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.60799999999999998,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":1.008},"last":0.67400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740652,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77900000000000003,"5min":0.78300000000000003,"15min":0.78500000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73699999999999999,"5min":0.78100000000000003,"15min":0.81100000000000005},"last":0.61299999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.47999999999999998,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76100000000000001,"5min":0.90800000000000003,"15min":1.046},"last":0.67200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359740655,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68100000000000005,"5min":0.72399999999999998,"15min":0.79200000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.47499999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.69099999999999995,"5min":0.70699999999999996,"15min":0.83899999999999997},"last":0.60599999999999998}]},{"osd":2,"last update":"Fri Jan 12 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.53800000000000003,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.64000000000000001},"min":{"1min":0.56999999999999995,"5min":0.48899999999999999,"15min":0.47499999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:54:06.623 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.623 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.623 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.623 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.623 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.624 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.624 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.624 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.624 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.624 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:06.624 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:54:06.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:06.927+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:07.295 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:07.295+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:07.901 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:07.899+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:08.321 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:08.319+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:08.914 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:08.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:09.300 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:09.299+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:09.882 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:09.879+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:10.275 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:10.271+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:10.862 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:10.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:11.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:11.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:11.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:11.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:12.328 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:12.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:12.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:12.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:13.304 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:13.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:13.840 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:13.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:14.311 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:14.311+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:14.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:14.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:15.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:15.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:15.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:15.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:16.315 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:16.315+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:16.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:16.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:17.290 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:17.287+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:17.828 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:17.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:18.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:18.239+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:18.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:18.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:19.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:19.243+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:19.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:19.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:20.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:20.211+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:20.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:20.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:21.225 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:21.223+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:21.849 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:21.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:22.223 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:22.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:22.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:22.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:23.210 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:23.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:23.809 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:23.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:24.164 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:24.163+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:24.835 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:24.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:25.173 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:25.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:25.851 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:25.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:26.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:26.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:26.625 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:54:26.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:26.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:26.924 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:54:26.926 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:54:26.937 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5866,"stamp":"2024-01-12T00:54:25.737874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003306"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1975,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891845+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:15.891845+0000","last_peered":"2024-01-11T22:48:15.891845+0000","last_clean":"2024-01-11T22:48:15.891845+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:15.891845+0000","last_undegraded":"2024-01-11T22:48:15.891845+0000","last_fullsized":"2024-01-11T22:48:15.891845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1708,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891554+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:15.891554+0000","last_peered":"2024-01-11T22:48:15.891554+0000","last_clean":"2024-01-11T22:48:15.891554+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:15.891554+0000","last_undegraded":"2024-01-11T22:48:15.891554+0000","last_fullsized":"2024-01-11T22:48:15.891554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1920,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891449+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:15.891449+0000","last_peered":"2024-01-11T22:48:15.891449+0000","last_clean":"2024-01-11T22:48:15.891449+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:15.891449+0000","last_undegraded":"2024-01-11T22:48:15.891449+0000","last_fullsized":"2024-01-11T22:48:15.891449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12866,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890524+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:15.890524+0000","last_peered":"2024-01-11T22:48:15.890524+0000","last_clean":"2024-01-11T22:48:15.890524+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:15.890524+0000","last_undegraded":"2024-01-11T22:48:15.890524+0000","last_fullsized":"2024-01-11T22:48:15.890524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2139,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815190+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815190+0000","last_peered":"2024-01-12T00:53:48.815190+0000","last_clean":"2024-01-12T00:53:48.815190+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815190+0000","last_undegraded":"2024-01-12T00:53:48.815190+0000","last_fullsized":"2024-01-12T00:53:48.815190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12350,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891077+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:15.891077+0000","last_peered":"2024-01-11T22:48:15.891077+0000","last_clean":"2024-01-11T22:48:15.891077+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:15.891077+0000","last_undegraded":"2024-01-11T22:48:15.891077+0000","last_fullsized":"2024-01-11T22:48:15.891077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7501,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890630+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:15.890630+0000","last_peered":"2024-01-11T22:48:15.890630+0000","last_clean":"2024-01-11T22:48:15.890630+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:15.890630+0000","last_undegraded":"2024-01-11T22:48:15.890630+0000","last_fullsized":"2024-01-11T22:48:15.890630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11778,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891091+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:15.891091+0000","last_peered":"2024-01-11T22:48:15.891091+0000","last_clean":"2024-01-11T22:48:15.891091+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:15.891091+0000","last_undegraded":"2024-01-11T22:48:15.891091+0000","last_fullsized":"2024-01-11T22:48:15.891091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891410+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:15.891410+0000","last_peered":"2024-01-11T22:48:15.891410+0000","last_clean":"2024-01-11T22:48:15.891410+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:15.891410+0000","last_undegraded":"2024-01-11T22:48:15.891410+0000","last_fullsized":"2024-01-11T22:48:15.891410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891107+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:15.891107+0000","last_peered":"2024-01-11T22:48:15.891107+0000","last_clean":"2024-01-11T22:48:15.891107+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:15.891107+0000","last_undegraded":"2024-01-11T22:48:15.891107+0000","last_fullsized":"2024-01-11T22:48:15.891107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740658,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.443,"5min":0.37,"15min":0.37},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.46400000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Jan 12 00:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.45700000000000002,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.60799999999999998,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":1.008},"last":0.71299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740656,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77900000000000003,"5min":0.78300000000000003,"15min":0.78500000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73699999999999999,"5min":0.78100000000000003,"15min":0.81100000000000005},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Jan 12 00:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.47999999999999998,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76100000000000001,"5min":0.90800000000000003,"15min":1.046},"last":0.66700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359740659,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.42599999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.83899999999999997},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Jan 12 00:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.73999999999999999,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.629},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75600000000000001,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.71499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:54:26.938 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.938 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.938 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.939 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.939 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.939 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.939 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.939 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.940 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.940 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:26.940 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:54:27.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:27.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:27.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:27.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:28.102 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:28.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:28.855 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:28.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:29.146 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:29.143+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:29.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:29.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:30.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:30.167+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:30.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:30.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:31.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:31.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:31.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:31.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:32.255 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:32.251+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:32.852 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:32.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:33.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:33.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:33.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:33.835+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:34.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:34.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:34.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:34.851+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:35.239 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:35.235+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:35.817 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:35.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:36.256 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:36.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:36.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:36.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:37.305 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:37.303+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:37.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:37.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:38.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:38.343+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:38.831 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:38.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:39.340 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:39.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:39.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:39.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:40.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:40.335+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:40.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:40.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:41.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:41.355+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:41.906 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:41.903+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:42.329 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:42.327+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:42.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:42.939+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:43.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:43.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:43.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:43.935+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:44.341 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:44.339+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:44.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:44.971+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:45.309 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:45.307+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:45.970 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:45.967+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:46.324 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:46.323+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:46.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:54:46.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:46.987+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:47.228 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:54:47.231 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:54:47.241 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5876,"stamp":"2024-01-12T00:54:45.743500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1975,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891845+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:15.891845+0000","last_peered":"2024-01-11T22:48:15.891845+0000","last_clean":"2024-01-11T22:48:15.891845+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:15.891845+0000","last_undegraded":"2024-01-11T22:48:15.891845+0000","last_fullsized":"2024-01-11T22:48:15.891845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1708,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891554+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:15.891554+0000","last_peered":"2024-01-11T22:48:15.891554+0000","last_clean":"2024-01-11T22:48:15.891554+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:15.891554+0000","last_undegraded":"2024-01-11T22:48:15.891554+0000","last_fullsized":"2024-01-11T22:48:15.891554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1920,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891449+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:15.891449+0000","last_peered":"2024-01-11T22:48:15.891449+0000","last_clean":"2024-01-11T22:48:15.891449+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:15.891449+0000","last_undegraded":"2024-01-11T22:48:15.891449+0000","last_fullsized":"2024-01-11T22:48:15.891449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12866,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890524+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:15.890524+0000","last_peered":"2024-01-11T22:48:15.890524+0000","last_clean":"2024-01-11T22:48:15.890524+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:15.890524+0000","last_undegraded":"2024-01-11T22:48:15.890524+0000","last_fullsized":"2024-01-11T22:48:15.890524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2139,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815190+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815190+0000","last_peered":"2024-01-12T00:53:48.815190+0000","last_clean":"2024-01-12T00:53:48.815190+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815190+0000","last_undegraded":"2024-01-12T00:53:48.815190+0000","last_fullsized":"2024-01-12T00:53:48.815190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12350,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891077+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:15.891077+0000","last_peered":"2024-01-11T22:48:15.891077+0000","last_clean":"2024-01-11T22:48:15.891077+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:15.891077+0000","last_undegraded":"2024-01-11T22:48:15.891077+0000","last_fullsized":"2024-01-11T22:48:15.891077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7501,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890630+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:15.890630+0000","last_peered":"2024-01-11T22:48:15.890630+0000","last_clean":"2024-01-11T22:48:15.890630+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:15.890630+0000","last_undegraded":"2024-01-11T22:48:15.890630+0000","last_fullsized":"2024-01-11T22:48:15.890630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11778,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891091+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:15.891091+0000","last_peered":"2024-01-11T22:48:15.891091+0000","last_clean":"2024-01-11T22:48:15.891091+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:15.891091+0000","last_undegraded":"2024-01-11T22:48:15.891091+0000","last_fullsized":"2024-01-11T22:48:15.891091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891410+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:15.891410+0000","last_peered":"2024-01-11T22:48:15.891410+0000","last_clean":"2024-01-11T22:48:15.891410+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:15.891410+0000","last_undegraded":"2024-01-11T22:48:15.891410+0000","last_fullsized":"2024-01-11T22:48:15.891410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891107+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:15.891107+0000","last_peered":"2024-01-11T22:48:15.891107+0000","last_clean":"2024-01-11T22:48:15.891107+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:15.891107+0000","last_undegraded":"2024-01-11T22:48:15.891107+0000","last_fullsized":"2024-01-11T22:48:15.891107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740662,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.52700000000000002},"min":{"1min":0.46200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.63600000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":1.028,"15min":1.028},"last":0.71599999999999997}]},{"osd":1,"last update":"Fri Jan 12 00:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.46100000000000002,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.74399999999999999,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69599999999999995,"15min":0.69499999999999995},"min":{"1min":0.57699999999999996,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":1.008},"last":0.82299999999999995}]}]},{"osd":1,"up_from":8,"seq":34359740660,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77900000000000003,"5min":0.78300000000000003,"15min":0.78500000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73699999999999999,"5min":0.78100000000000003,"15min":0.81100000000000005},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Jan 12 00:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.53500000000000003,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.71999999999999997,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67300000000000004,"15min":0.67000000000000004},"min":{"1min":0.55500000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.67500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359740663,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.42599999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.83899999999999997},"last":0.63600000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.73999999999999999,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75600000000000001,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.67500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:54:47.242 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.242 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.243 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.243 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.243 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.243 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.243 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.244 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.244 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.244 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:54:47.244 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:54:47.278 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:47.275+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:47.978 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:47.974+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:48.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:48.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:49.002 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:48.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:49.310 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:49.306+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:49.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:49.962+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:50.314 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:50.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:50.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:50.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:51.322 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:51.318+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:51.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:51.950+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:52.292 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:52.290+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:52.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:52.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:53.262 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:53.258+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:54.018 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:54.014+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:54.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:54.302+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:54.989 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:54.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:55.266 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:55.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:56.012 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:56.010+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:56.219 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:56.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:57.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:56.998+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:57.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:57.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:57.979 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:57.978+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:58.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:58.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:58.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:58.950+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:59.160 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:54:59.158+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:54:59.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:54:59.958+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:00.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:00.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:00.991 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:00.986+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:01.143 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:01.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:01.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:01.982+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:02.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:02.138+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:02.993 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:02.990+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:03.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:03.170+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:04.024 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:04.022+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:04.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:04.122+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:05.009 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:05.006+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:05.103 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:05.102+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:06.039 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:06.038+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:06.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:06.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:07.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:07.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:07.064 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:07.062+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:07.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:55:07.540 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:55:07.543 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:55:07.554 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5886,"stamp":"2024-01-12T00:55:05.748917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1975,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891845+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:15.891845+0000","last_peered":"2024-01-11T22:48:15.891845+0000","last_clean":"2024-01-11T22:48:15.891845+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:15.891845+0000","last_undegraded":"2024-01-11T22:48:15.891845+0000","last_fullsized":"2024-01-11T22:48:15.891845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1708,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891554+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:15.891554+0000","last_peered":"2024-01-11T22:48:15.891554+0000","last_clean":"2024-01-11T22:48:15.891554+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:15.891554+0000","last_undegraded":"2024-01-11T22:48:15.891554+0000","last_fullsized":"2024-01-11T22:48:15.891554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1920,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891449+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:15.891449+0000","last_peered":"2024-01-11T22:48:15.891449+0000","last_clean":"2024-01-11T22:48:15.891449+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:15.891449+0000","last_undegraded":"2024-01-11T22:48:15.891449+0000","last_fullsized":"2024-01-11T22:48:15.891449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12866,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890524+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:15.890524+0000","last_peered":"2024-01-11T22:48:15.890524+0000","last_clean":"2024-01-11T22:48:15.890524+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:15.890524+0000","last_undegraded":"2024-01-11T22:48:15.890524+0000","last_fullsized":"2024-01-11T22:48:15.890524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2139,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815190+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815190+0000","last_peered":"2024-01-12T00:53:48.815190+0000","last_clean":"2024-01-12T00:53:48.815190+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815190+0000","last_undegraded":"2024-01-12T00:53:48.815190+0000","last_fullsized":"2024-01-12T00:53:48.815190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12350,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891077+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:15.891077+0000","last_peered":"2024-01-11T22:48:15.891077+0000","last_clean":"2024-01-11T22:48:15.891077+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:15.891077+0000","last_undegraded":"2024-01-11T22:48:15.891077+0000","last_fullsized":"2024-01-11T22:48:15.891077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7501,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890630+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:15.890630+0000","last_peered":"2024-01-11T22:48:15.890630+0000","last_clean":"2024-01-11T22:48:15.890630+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:15.890630+0000","last_undegraded":"2024-01-11T22:48:15.890630+0000","last_fullsized":"2024-01-11T22:48:15.890630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11778,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891091+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:15.891091+0000","last_peered":"2024-01-11T22:48:15.891091+0000","last_clean":"2024-01-11T22:48:15.891091+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:15.891091+0000","last_undegraded":"2024-01-11T22:48:15.891091+0000","last_fullsized":"2024-01-11T22:48:15.891091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891410+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:15.891410+0000","last_peered":"2024-01-11T22:48:15.891410+0000","last_clean":"2024-01-11T22:48:15.891410+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:15.891410+0000","last_undegraded":"2024-01-11T22:48:15.891410+0000","last_fullsized":"2024-01-11T22:48:15.891410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891107+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:15.891107+0000","last_peered":"2024-01-11T22:48:15.891107+0000","last_clean":"2024-01-11T22:48:15.891107+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:15.891107+0000","last_undegraded":"2024-01-11T22:48:15.891107+0000","last_fullsized":"2024-01-11T22:48:15.891107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740666,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.52700000000000002},"min":{"1min":0.46200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.63600000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":1.028,"15min":1.028},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Jan 12 00:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.46100000000000002,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.74399999999999999,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69599999999999995,"15min":0.69499999999999995},"min":{"1min":0.57699999999999996,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":1.008},"last":0.71199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740664,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.78300000000000003,"15min":0.78500000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.47099999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.53500000000000003,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.71999999999999997,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67300000000000004,"15min":0.67000000000000004},"min":{"1min":0.55500000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.67500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359740667,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.42599999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.83899999999999997},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Jan 12 00:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.73999999999999999,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.623},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75600000000000001,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:55:07.555 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.556 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.556 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.556 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.556 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.556 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.557 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.557 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.557 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.557 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:07.557 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:55:08.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:08.066+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:08.083 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:08.082+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:09.031 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:09.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:09.082 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:09.078+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:10.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:10.010+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:10.117 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:10.114+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:10.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:10.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:11.104 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:11.102+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:11.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:11.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:12.144 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:12.142+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:13.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:13.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:13.151 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:13.150+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:14.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:14.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:14.148 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:14.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:15.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:15.022+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:15.198 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:15.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:16.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:16.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:16.209 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:16.206+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:17.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:17.030+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:17.205 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:17.202+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:17.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:17.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:18.165 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:18.162+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:18.972 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:18.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:19.175 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:19.174+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:19.936 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:19.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:20.129 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:20.126+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:20.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:20.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:21.150 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:21.146+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:21.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:21.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:22.178 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:22.174+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:22.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:22.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:23.189 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:23.186+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:23.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:23.922+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:24.214 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:24.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:24.970 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:24.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:25.193 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:25.190+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:25.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:25.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:26.232 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:26.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:26.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:26.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:27.239 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:27.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:27.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:55:27.850 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:55:27.852 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:55:27.864 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5897,"stamp":"2024-01-12T00:55:27.754904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003356"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1975,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891845+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:15.891845+0000","last_peered":"2024-01-11T22:48:15.891845+0000","last_clean":"2024-01-11T22:48:15.891845+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:15.891845+0000","last_undegraded":"2024-01-11T22:48:15.891845+0000","last_fullsized":"2024-01-11T22:48:15.891845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1708,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891554+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:15.891554+0000","last_peered":"2024-01-11T22:48:15.891554+0000","last_clean":"2024-01-11T22:48:15.891554+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:15.891554+0000","last_undegraded":"2024-01-11T22:48:15.891554+0000","last_fullsized":"2024-01-11T22:48:15.891554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1920,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891449+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:15.891449+0000","last_peered":"2024-01-11T22:48:15.891449+0000","last_clean":"2024-01-11T22:48:15.891449+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:15.891449+0000","last_undegraded":"2024-01-11T22:48:15.891449+0000","last_fullsized":"2024-01-11T22:48:15.891449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12866,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890524+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:15.890524+0000","last_peered":"2024-01-11T22:48:15.890524+0000","last_clean":"2024-01-11T22:48:15.890524+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:15.890524+0000","last_undegraded":"2024-01-11T22:48:15.890524+0000","last_fullsized":"2024-01-11T22:48:15.890524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2139,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815190+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815190+0000","last_peered":"2024-01-12T00:53:48.815190+0000","last_clean":"2024-01-12T00:53:48.815190+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815190+0000","last_undegraded":"2024-01-12T00:53:48.815190+0000","last_fullsized":"2024-01-12T00:53:48.815190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12350,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891077+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:15.891077+0000","last_peered":"2024-01-11T22:48:15.891077+0000","last_clean":"2024-01-11T22:48:15.891077+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:15.891077+0000","last_undegraded":"2024-01-11T22:48:15.891077+0000","last_fullsized":"2024-01-11T22:48:15.891077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7501,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890630+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:15.890630+0000","last_peered":"2024-01-11T22:48:15.890630+0000","last_clean":"2024-01-11T22:48:15.890630+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:15.890630+0000","last_undegraded":"2024-01-11T22:48:15.890630+0000","last_fullsized":"2024-01-11T22:48:15.890630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11778,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891091+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:15.891091+0000","last_peered":"2024-01-11T22:48:15.891091+0000","last_clean":"2024-01-11T22:48:15.891091+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:15.891091+0000","last_undegraded":"2024-01-11T22:48:15.891091+0000","last_fullsized":"2024-01-11T22:48:15.891091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891410+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:15.891410+0000","last_peered":"2024-01-11T22:48:15.891410+0000","last_clean":"2024-01-11T22:48:15.891410+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:15.891410+0000","last_undegraded":"2024-01-11T22:48:15.891410+0000","last_fullsized":"2024-01-11T22:48:15.891410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891107+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:15.891107+0000","last_peered":"2024-01-11T22:48:15.891107+0000","last_clean":"2024-01-11T22:48:15.891107+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:15.891107+0000","last_undegraded":"2024-01-11T22:48:15.891107+0000","last_fullsized":"2024-01-11T22:48:15.891107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740670,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.52700000000000002},"min":{"1min":0.46200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.63600000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":1.028,"15min":1.028},"last":0.503}]},{"osd":1,"last update":"Fri Jan 12 00:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.46100000000000002,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.74399999999999999,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69599999999999995,"15min":0.69499999999999995},"min":{"1min":0.57699999999999996,"5min":0.53900000000000003,"15min":0.51900000000000002},"max":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":1.008},"last":0.60799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740668,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.78300000000000003,"15min":0.78500000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.47099999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.74299999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.53500000000000003,"5min":0.45700000000000002,"15min":0.38600000000000001},"max":{"1min":0.71999999999999997,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67300000000000004,"15min":0.67000000000000004},"min":{"1min":0.55500000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.66700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359740671,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.54700000000000004},"min":{"1min":0.46400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.503,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.64100000000000001,"5min":0.70299999999999996,"15min":0.83899999999999997},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64100000000000001,"15min":0.63900000000000001},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.73299999999999998,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:55:27.865 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.866 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.866 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.866 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.866 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.866 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.866 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.867 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.867 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.867 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:27.867 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:55:27.946 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:27.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:28.231 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:28.226+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:28.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:28.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:29.197 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:29.194+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:29.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:29.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:30.164 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:30.162+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:30.977 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:30.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:31.173 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:31.170+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:31.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:31.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:32.218 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:32.214+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:32.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:32.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:33.219 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:33.218+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:33.954 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:33.950+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:34.228 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:34.226+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:34.958 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:34.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:35.253 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:35.250+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:35.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:35.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:36.259 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:36.258+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:36.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:36.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:37.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:37.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:38.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:38.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:38.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:38.290+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:39.042 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:39.038+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:39.275 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:39.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:40.085 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:40.082+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:40.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:40.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:41.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:41.041+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:41.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:41.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:42.018 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:42.013+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:42.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:42.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:42.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:42.985+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:43.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:43.313+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:43.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:43.985+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:44.319 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:44.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:44.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:44.989+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:45.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:45.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:46.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:46.001+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:46.245 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:46.241+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:47.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:47.001+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:47.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:47.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:47.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:55:48.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:48.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:48.166 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:55:48.169 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:55:48.179 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5907,"stamp":"2024-01-12T00:55:47.760229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003238"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1975,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891845+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:15.891845+0000","last_peered":"2024-01-11T22:48:15.891845+0000","last_clean":"2024-01-11T22:48:15.891845+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:15.891845+0000","last_undegraded":"2024-01-11T22:48:15.891845+0000","last_fullsized":"2024-01-11T22:48:15.891845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1708,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891554+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:15.891554+0000","last_peered":"2024-01-11T22:48:15.891554+0000","last_clean":"2024-01-11T22:48:15.891554+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:15.891554+0000","last_undegraded":"2024-01-11T22:48:15.891554+0000","last_fullsized":"2024-01-11T22:48:15.891554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1920,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891449+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:15.891449+0000","last_peered":"2024-01-11T22:48:15.891449+0000","last_clean":"2024-01-11T22:48:15.891449+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:15.891449+0000","last_undegraded":"2024-01-11T22:48:15.891449+0000","last_fullsized":"2024-01-11T22:48:15.891449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12866,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890524+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:15.890524+0000","last_peered":"2024-01-11T22:48:15.890524+0000","last_clean":"2024-01-11T22:48:15.890524+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:15.890524+0000","last_undegraded":"2024-01-11T22:48:15.890524+0000","last_fullsized":"2024-01-11T22:48:15.890524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2139,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815190+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815190+0000","last_peered":"2024-01-12T00:53:48.815190+0000","last_clean":"2024-01-12T00:53:48.815190+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815190+0000","last_undegraded":"2024-01-12T00:53:48.815190+0000","last_fullsized":"2024-01-12T00:53:48.815190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12350,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891077+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:15.891077+0000","last_peered":"2024-01-11T22:48:15.891077+0000","last_clean":"2024-01-11T22:48:15.891077+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:15.891077+0000","last_undegraded":"2024-01-11T22:48:15.891077+0000","last_fullsized":"2024-01-11T22:48:15.891077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7501,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890630+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:15.890630+0000","last_peered":"2024-01-11T22:48:15.890630+0000","last_clean":"2024-01-11T22:48:15.890630+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:15.890630+0000","last_undegraded":"2024-01-11T22:48:15.890630+0000","last_fullsized":"2024-01-11T22:48:15.890630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11778,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891091+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:15.891091+0000","last_peered":"2024-01-11T22:48:15.891091+0000","last_clean":"2024-01-11T22:48:15.891091+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:15.891091+0000","last_undegraded":"2024-01-11T22:48:15.891091+0000","last_fullsized":"2024-01-11T22:48:15.891091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891410+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:15.891410+0000","last_peered":"2024-01-11T22:48:15.891410+0000","last_clean":"2024-01-11T22:48:15.891410+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:15.891410+0000","last_undegraded":"2024-01-11T22:48:15.891410+0000","last_fullsized":"2024-01-11T22:48:15.891410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891107+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:15.891107+0000","last_peered":"2024-01-11T22:48:15.891107+0000","last_clean":"2024-01-11T22:48:15.891107+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:15.891107+0000","last_undegraded":"2024-01-11T22:48:15.891107+0000","last_fullsized":"2024-01-11T22:48:15.891107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740674,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.64600000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.47699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71599999999999997,"5min":1.028,"15min":1.028},"last":0.47599999999999998}]},{"osd":1,"last update":"Fri Jan 12 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.44800000000000001,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.77500000000000002,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.69199999999999995,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.51900000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.008},"last":0.65200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740672,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.78300000000000003,"15min":0.78500000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.47099999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.64300000000000002}]},{"osd":2,"last update":"Fri Jan 12 00:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.38600000000000001},"max":{"1min":0.65700000000000003,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.70799999999999996,"5min":1.071,"15min":1.071},"last":0.72899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359740675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.54700000000000004},"min":{"1min":0.46400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.625},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.503,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.64100000000000001,"5min":0.70299999999999996,"15min":0.83899999999999997},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Jan 12 00:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64100000000000001,"15min":0.63900000000000001},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.73299999999999998,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.71899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:55:48.180 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.180 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.180 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.181 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.181 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.181 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.181 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.181 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.182 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.182 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:55:48.182 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:55:48.293 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:48.289+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:49.011 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:49.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:49.335 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:49.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:50.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:50.009+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:50.313 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:50.309+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:51.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:51.021+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:51.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:51.265+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:52.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:52.037+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:52.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:52.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:53.028 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:53.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:53.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:53.277+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:54.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:54.065+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:54.298 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:54.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:55.087 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:55.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:55.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:55.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:56.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:56.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:56.329 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:56.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:57.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:57.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:57.301 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:57.297+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:58.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:58.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:58.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:58.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:59.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:55:59.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:55:59.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:55:59.341+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:00.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:00.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:00.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:00.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:01.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:01.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:01.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:01.345+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:02.112 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:02.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:02.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:02.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:03.135 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:03.133+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:03.277 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:03.273+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:04.161 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:04.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:04.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:04.297+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:05.139 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:05.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:05.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:05.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:06.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:06.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:06.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:06.341+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:07.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:07.105+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:07.320 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:07.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:08.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:08.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:08.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:56:08.334 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:08.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:08.477 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:56:08.481 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:56:08.491 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5917,"stamp":"2024-01-12T00:56:07.765692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003258"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1975,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891845+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-11T22:48:15.891845+0000","last_peered":"2024-01-11T22:48:15.891845+0000","last_clean":"2024-01-11T22:48:15.891845+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-11T22:48:15.891845+0000","last_undegraded":"2024-01-11T22:48:15.891845+0000","last_fullsized":"2024-01-11T22:48:15.891845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1708,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891554+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-11T22:48:15.891554+0000","last_peered":"2024-01-11T22:48:15.891554+0000","last_clean":"2024-01-11T22:48:15.891554+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-11T22:48:15.891554+0000","last_undegraded":"2024-01-11T22:48:15.891554+0000","last_fullsized":"2024-01-11T22:48:15.891554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1920,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891449+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-11T22:48:15.891449+0000","last_peered":"2024-01-11T22:48:15.891449+0000","last_clean":"2024-01-11T22:48:15.891449+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-11T22:48:15.891449+0000","last_undegraded":"2024-01-11T22:48:15.891449+0000","last_fullsized":"2024-01-11T22:48:15.891449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12866,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890524+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-11T22:48:15.890524+0000","last_peered":"2024-01-11T22:48:15.890524+0000","last_clean":"2024-01-11T22:48:15.890524+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-11T22:48:15.890524+0000","last_undegraded":"2024-01-11T22:48:15.890524+0000","last_fullsized":"2024-01-11T22:48:15.890524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2139,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815190+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815190+0000","last_peered":"2024-01-12T00:53:48.815190+0000","last_clean":"2024-01-12T00:53:48.815190+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815190+0000","last_undegraded":"2024-01-12T00:53:48.815190+0000","last_fullsized":"2024-01-12T00:53:48.815190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12350,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891077+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-11T22:48:15.891077+0000","last_peered":"2024-01-11T22:48:15.891077+0000","last_clean":"2024-01-11T22:48:15.891077+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-11T22:48:15.891077+0000","last_undegraded":"2024-01-11T22:48:15.891077+0000","last_fullsized":"2024-01-11T22:48:15.891077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7501,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.890630+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-11T22:48:15.890630+0000","last_peered":"2024-01-11T22:48:15.890630+0000","last_clean":"2024-01-11T22:48:15.890630+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-11T22:48:15.890630+0000","last_undegraded":"2024-01-11T22:48:15.890630+0000","last_fullsized":"2024-01-11T22:48:15.890630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11778,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891091+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-11T22:48:15.891091+0000","last_peered":"2024-01-11T22:48:15.891091+0000","last_clean":"2024-01-11T22:48:15.891091+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-11T22:48:15.891091+0000","last_undegraded":"2024-01-11T22:48:15.891091+0000","last_fullsized":"2024-01-11T22:48:15.891091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891410+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-11T22:48:15.891410+0000","last_peered":"2024-01-11T22:48:15.891410+0000","last_clean":"2024-01-11T22:48:15.891410+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-11T22:48:15.891410+0000","last_undegraded":"2024-01-11T22:48:15.891410+0000","last_fullsized":"2024-01-11T22:48:15.891410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-11T22:48:15.891107+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-11T22:48:15.891107+0000","last_peered":"2024-01-11T22:48:15.891107+0000","last_clean":"2024-01-11T22:48:15.891107+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-11T22:48:15.891107+0000","last_undegraded":"2024-01-11T22:48:15.891107+0000","last_fullsized":"2024-01-11T22:48:15.891107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740678,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.64600000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.503},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.47699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71599999999999997,"5min":1.028,"15min":1.028},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Jan 12 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.44800000000000001,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.77500000000000002,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.69199999999999995,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.51900000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.008},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740676,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.505,"15min":0.52000000000000002},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.56999999999999995,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.622},"min":{"1min":0.499,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Jan 12 00:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.38600000000000001},"max":{"1min":0.65700000000000003,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.70799999999999996,"5min":1.071,"15min":1.071},"last":0.61099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.54700000000000004},"min":{"1min":0.46400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.503,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.64100000000000001,"5min":0.70299999999999996,"15min":0.83899999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Jan 12 00:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64100000000000001,"15min":0.63900000000000001},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.73299999999999998,"5min":0.76500000000000001,"15min":0.95199999999999996},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:56:08.493 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.493 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.494 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.494 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.494 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.494 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.495 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.495 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.495 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.495 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:08.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2024-01-12T00:56:08.783 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2024-01-12T00:56:08.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2024-01-12T00:56:09.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:09.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:09.070 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2024-01-12T00:56:09.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2024-01-12T00:56:09.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:09.309+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:09.366 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2024-01-12T00:56:09.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2024-01-12T00:56:09.660 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2024-01-12T00:56:09.675 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2024-01-12T00:56:09.960 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2024-01-12T00:56:09.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2024-01-12T00:56:10.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:10.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:10.260 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-01-12T00:56:10.274 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2024-01-12T00:56:10.304 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:10.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:10.563 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2024-01-12T00:56:10.573 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2024-01-12T00:56:10.850 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2024-01-12T00:56:10.860 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2024-01-12T00:56:11.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:11.085+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:11.152 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2024-01-12T00:56:11.163 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2024-01-12T00:56:11.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:11.269+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:11.451 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-01-12T00:56:11.461 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:56:12.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:12.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:12.280 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:12.277+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:13.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:13.093+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:13.297 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:13.293+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:14.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:14.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:14.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:14.257+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:15.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:15.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:15.282 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:15.277+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:16.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:16.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:16.262 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:16.257+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:17.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:17.025+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:17.286 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:17.281+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:18.060 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:18.057+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:18.292 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:18.289+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:19.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:19.101+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:19.288 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:19.285+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:20.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:20.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:20.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:20.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:21.113 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:21.109+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:21.303 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:21.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:22.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:22.065+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:22.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:22.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:23.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:23.077+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:23.336 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:23.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:24.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:24.089+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:24.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:24.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:25.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:25.125+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:25.355 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:25.353+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:26.141 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:26.137+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:26.395 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:26.393+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:27.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:27.097+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:27.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:27.381+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:28.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:28.145+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:28.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:28.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:29.168 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:29.165+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:29.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:29.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:30.145 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:30.141+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:30.485 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:30.481+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:31.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:31.113+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:31.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:56:31.478 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:31.473+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:31.755 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:56:31.757 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:56:31.767 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5928,"stamp":"2024-01-12T00:56:29.771687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003359"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1976,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.366550+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:53:43.366550+0000","last_peered":"2024-01-12T00:53:43.366550+0000","last_clean":"2024-01-12T00:53:43.366550+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:53:43.366550+0000","last_undegraded":"2024-01-12T00:53:43.366550+0000","last_fullsized":"2024-01-12T00:53:43.366550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1709,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.663055+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:53:43.663055+0000","last_peered":"2024-01-12T00:53:43.663055+0000","last_clean":"2024-01-12T00:53:43.663055+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:53:43.663055+0000","last_undegraded":"2024-01-12T00:53:43.663055+0000","last_fullsized":"2024-01-12T00:53:43.663055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1921,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.961739+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:53:43.961739+0000","last_peered":"2024-01-12T00:53:43.961739+0000","last_clean":"2024-01-12T00:53:43.961739+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:53:43.961739+0000","last_undegraded":"2024-01-12T00:53:43.961739+0000","last_fullsized":"2024-01-12T00:53:43.961739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12867,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:44.543961+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:53:44.543961+0000","last_peered":"2024-01-12T00:53:44.543961+0000","last_clean":"2024-01-12T00:53:44.543961+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:53:44.543961+0000","last_undegraded":"2024-01-12T00:53:44.543961+0000","last_fullsized":"2024-01-12T00:53:44.543961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2140,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815444+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815444+0000","last_peered":"2024-01-12T00:53:48.815444+0000","last_clean":"2024-01-12T00:53:48.815444+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815444+0000","last_undegraded":"2024-01-12T00:53:48.815444+0000","last_fullsized":"2024-01-12T00:53:48.815444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12351,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.132240+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:53:45.132240+0000","last_peered":"2024-01-12T00:53:45.132240+0000","last_clean":"2024-01-12T00:53:45.132240+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:53:45.132240+0000","last_undegraded":"2024-01-12T00:53:45.132240+0000","last_fullsized":"2024-01-12T00:53:45.132240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7502,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.431908+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:53:45.431908+0000","last_peered":"2024-01-12T00:53:45.431908+0000","last_clean":"2024-01-12T00:53:45.431908+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:53:45.431908+0000","last_undegraded":"2024-01-12T00:53:45.431908+0000","last_fullsized":"2024-01-12T00:53:45.431908+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11779,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.720198+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:53:45.720198+0000","last_peered":"2024-01-12T00:53:45.720198+0000","last_clean":"2024-01-12T00:53:45.720198+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:53:45.720198+0000","last_undegraded":"2024-01-12T00:53:45.720198+0000","last_fullsized":"2024-01-12T00:53:45.720198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2517,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.016771+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:53:46.016771+0000","last_peered":"2024-01-12T00:53:46.016771+0000","last_clean":"2024-01-12T00:53:46.016771+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:53:46.016771+0000","last_undegraded":"2024-01-12T00:53:46.016771+0000","last_fullsized":"2024-01-12T00:53:46.016771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.312259+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:53:46.312259+0000","last_peered":"2024-01-12T00:53:46.312259+0000","last_clean":"2024-01-12T00:53:46.312259+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:53:46.312259+0000","last_undegraded":"2024-01-12T00:53:46.312259+0000","last_fullsized":"2024-01-12T00:53:46.312259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740682,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54100000000000004,"15min":0.52800000000000002},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.64600000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.47699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71599999999999997,"5min":1.028,"15min":1.028},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Jan 12 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.44800000000000001,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.77500000000000002,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.69199999999999995,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.51900000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.008},"last":0.76000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740680,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.505,"15min":0.52000000000000002},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.56999999999999995,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.441},{"interface":"front","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.622},"min":{"1min":0.499,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.51400000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.38600000000000001},"max":{"1min":0.65700000000000003,"5min":0.75600000000000001,"15min":0.97299999999999998},"last":0.62},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.55300000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.70799999999999996,"5min":1.071,"15min":1.071},"last":0.57199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359740683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.46300000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.49299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.83899999999999997},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Jan 12 00:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.66900000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.55800000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.749,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:56:31.769 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.769 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.769 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.769 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.769 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.770 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.770 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.770 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.770 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.770 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:31.770 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:56:32.126 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:32.121+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:32.446 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:32.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:33.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:33.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:33.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:33.476+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:34.071 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:34.068+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:34.452 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:34.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:35.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:35.068+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:35.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:35.492+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:36.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:36.072+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:36.483 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:36.480+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:37.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:37.088+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:37.450 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:37.444+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:38.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:38.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:38.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:38.476+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:39.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:39.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:39.472 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:39.468+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:40.134 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:40.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:40.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:40.488+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:41.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:41.168+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:41.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:41.440+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:42.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:42.208+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:42.397 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:42.392+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:43.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:43.204+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:43.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:43.404+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:44.165 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:44.160+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:44.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:44.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:45.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:45.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:45.465 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:45.460+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:46.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:46.188+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:46.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:46.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:47.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:47.196+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:47.402 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:47.396+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:48.180 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:48.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:48.404 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:48.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:49.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:49.156+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:49.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:49.384+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:50.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:50.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:50.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:50.356+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:51.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:51.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:51.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:51.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:51.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:56:52.063 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:56:52.066 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:56:52.076 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5939,"stamp":"2024-01-12T00:56:51.778072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003465"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1976,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.366550+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:53:43.366550+0000","last_peered":"2024-01-12T00:53:43.366550+0000","last_clean":"2024-01-12T00:53:43.366550+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:53:43.366550+0000","last_undegraded":"2024-01-12T00:53:43.366550+0000","last_fullsized":"2024-01-12T00:53:43.366550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1709,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.663055+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:53:43.663055+0000","last_peered":"2024-01-12T00:53:43.663055+0000","last_clean":"2024-01-12T00:53:43.663055+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:53:43.663055+0000","last_undegraded":"2024-01-12T00:53:43.663055+0000","last_fullsized":"2024-01-12T00:53:43.663055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1921,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.961739+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:53:43.961739+0000","last_peered":"2024-01-12T00:53:43.961739+0000","last_clean":"2024-01-12T00:53:43.961739+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:53:43.961739+0000","last_undegraded":"2024-01-12T00:53:43.961739+0000","last_fullsized":"2024-01-12T00:53:43.961739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12867,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:44.543961+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:53:44.543961+0000","last_peered":"2024-01-12T00:53:44.543961+0000","last_clean":"2024-01-12T00:53:44.543961+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:53:44.543961+0000","last_undegraded":"2024-01-12T00:53:44.543961+0000","last_fullsized":"2024-01-12T00:53:44.543961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2140,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815444+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815444+0000","last_peered":"2024-01-12T00:53:48.815444+0000","last_clean":"2024-01-12T00:53:48.815444+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815444+0000","last_undegraded":"2024-01-12T00:53:48.815444+0000","last_fullsized":"2024-01-12T00:53:48.815444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12351,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.132240+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:53:45.132240+0000","last_peered":"2024-01-12T00:53:45.132240+0000","last_clean":"2024-01-12T00:53:45.132240+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:53:45.132240+0000","last_undegraded":"2024-01-12T00:53:45.132240+0000","last_fullsized":"2024-01-12T00:53:45.132240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7502,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.431908+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:53:45.431908+0000","last_peered":"2024-01-12T00:53:45.431908+0000","last_clean":"2024-01-12T00:53:45.431908+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:53:45.431908+0000","last_undegraded":"2024-01-12T00:53:45.431908+0000","last_fullsized":"2024-01-12T00:53:45.431908+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11779,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.720198+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:53:45.720198+0000","last_peered":"2024-01-12T00:53:45.720198+0000","last_clean":"2024-01-12T00:53:45.720198+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:53:45.720198+0000","last_undegraded":"2024-01-12T00:53:45.720198+0000","last_fullsized":"2024-01-12T00:53:45.720198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2517,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.016771+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:53:46.016771+0000","last_peered":"2024-01-12T00:53:46.016771+0000","last_clean":"2024-01-12T00:53:46.016771+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:53:46.016771+0000","last_undegraded":"2024-01-12T00:53:46.016771+0000","last_fullsized":"2024-01-12T00:53:46.016771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.312259+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:53:46.312259+0000","last_peered":"2024-01-12T00:53:46.312259+0000","last_clean":"2024-01-12T00:53:46.312259+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:53:46.312259+0000","last_undegraded":"2024-01-12T00:53:46.312259+0000","last_fullsized":"2024-01-12T00:53:46.312259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740687,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.45600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.69499999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.47599999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64300000000000002,"5min":1.028,"15min":1.028},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 12 00:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.69099999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78200000000000003,"5min":0.82299999999999995,"15min":1.008},"last":0.65700000000000003}]}]},{"osd":1,"up_from":8,"seq":34359740685,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.505,"15min":0.52000000000000002},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.56999999999999995,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.622},"min":{"1min":0.499,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.69999999999999996}]},{"osd":2,"last update":"Fri Jan 12 00:56:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.38600000000000001},"max":{"1min":0.69199999999999995,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":1.071,"15min":1.071},"last":0.60899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.46300000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.49299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.83899999999999997},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Jan 12 00:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.66900000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.55800000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.749,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:56:52.077 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.078 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.078 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.078 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.078 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.078 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.078 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.079 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.079 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.079 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:56:52.079 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:56:52.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:52.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:52.283 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:52.280+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:53.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:53.056+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:53.248 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:53.244+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:54.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:54.072+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:54.200 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:54.196+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:55.089 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:55.084+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:55.236 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:55.232+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:56.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:56.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:56.216 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:56.212+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:57.127 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:57.120+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:57.226 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:57.220+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:58.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:58.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:58.195 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:58.188+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:59.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:56:59.200+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:56:59.243 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:56:59.240+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:00.176 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:00.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:00.242 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:00.236+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:01.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:01.156+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:01.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:01.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:02.175 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:02.172+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:02.255 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:02.252+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:03.181 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:03.176+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:03.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:03.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:04.152 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:04.148+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:04.273 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:04.268+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:05.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:05.112+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:05.265 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:05.260+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:06.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:06.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:06.278 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:06.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:07.110 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:07.104+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:07.321 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:07.316+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:08.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:08.060+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:08.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:08.352+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:09.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:09.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:09.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:09.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:10.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:10.020+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:10.316 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:10.312+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:11.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:11.056+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:11.349 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:11.348+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:12.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:12.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:12.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:57:12.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:12.340+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:12.376 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:57:12.379 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:57:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5949,"stamp":"2024-01-12T00:57:11.783522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1976,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.366550+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:53:43.366550+0000","last_peered":"2024-01-12T00:53:43.366550+0000","last_clean":"2024-01-12T00:53:43.366550+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:53:43.366550+0000","last_undegraded":"2024-01-12T00:53:43.366550+0000","last_fullsized":"2024-01-12T00:53:43.366550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1709,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.663055+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:53:43.663055+0000","last_peered":"2024-01-12T00:53:43.663055+0000","last_clean":"2024-01-12T00:53:43.663055+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:53:43.663055+0000","last_undegraded":"2024-01-12T00:53:43.663055+0000","last_fullsized":"2024-01-12T00:53:43.663055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1921,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.961739+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:53:43.961739+0000","last_peered":"2024-01-12T00:53:43.961739+0000","last_clean":"2024-01-12T00:53:43.961739+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:53:43.961739+0000","last_undegraded":"2024-01-12T00:53:43.961739+0000","last_fullsized":"2024-01-12T00:53:43.961739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12867,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:44.543961+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:53:44.543961+0000","last_peered":"2024-01-12T00:53:44.543961+0000","last_clean":"2024-01-12T00:53:44.543961+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:53:44.543961+0000","last_undegraded":"2024-01-12T00:53:44.543961+0000","last_fullsized":"2024-01-12T00:53:44.543961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2140,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815444+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815444+0000","last_peered":"2024-01-12T00:53:48.815444+0000","last_clean":"2024-01-12T00:53:48.815444+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815444+0000","last_undegraded":"2024-01-12T00:53:48.815444+0000","last_fullsized":"2024-01-12T00:53:48.815444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12351,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.132240+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:53:45.132240+0000","last_peered":"2024-01-12T00:53:45.132240+0000","last_clean":"2024-01-12T00:53:45.132240+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:53:45.132240+0000","last_undegraded":"2024-01-12T00:53:45.132240+0000","last_fullsized":"2024-01-12T00:53:45.132240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7502,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.431908+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:53:45.431908+0000","last_peered":"2024-01-12T00:53:45.431908+0000","last_clean":"2024-01-12T00:53:45.431908+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:53:45.431908+0000","last_undegraded":"2024-01-12T00:53:45.431908+0000","last_fullsized":"2024-01-12T00:53:45.431908+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11779,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.720198+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:53:45.720198+0000","last_peered":"2024-01-12T00:53:45.720198+0000","last_clean":"2024-01-12T00:53:45.720198+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:53:45.720198+0000","last_undegraded":"2024-01-12T00:53:45.720198+0000","last_fullsized":"2024-01-12T00:53:45.720198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2517,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.016771+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:53:46.016771+0000","last_peered":"2024-01-12T00:53:46.016771+0000","last_clean":"2024-01-12T00:53:46.016771+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:53:46.016771+0000","last_undegraded":"2024-01-12T00:53:46.016771+0000","last_fullsized":"2024-01-12T00:53:46.016771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.312259+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:53:46.312259+0000","last_peered":"2024-01-12T00:53:46.312259+0000","last_clean":"2024-01-12T00:53:46.312259+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:53:46.312259+0000","last_undegraded":"2024-01-12T00:53:46.312259+0000","last_fullsized":"2024-01-12T00:53:46.312259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740691,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.45600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.69499999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.47599999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64300000000000002,"5min":1.028,"15min":1.028},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jan 12 00:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.504},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.69099999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78200000000000003,"5min":0.82299999999999995,"15min":1.008},"last":0.63100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740689,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.42699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68999999999999995,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.40400000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Jan 12 00:56:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.38600000000000001},"max":{"1min":0.69199999999999995,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":1.071,"15min":1.071},"last":0.59599999999999997}]}]},{"osd":0,"up_from":8,"seq":34359740692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.46300000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.49299999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.83899999999999997},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.441},"max":{"1min":0.66900000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.55800000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.749,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:57:12.390 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.390 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.390 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.390 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.391 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.391 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.391 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.391 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.391 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.391 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:12.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:57:13.021 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:13.020+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:13.392 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:13.392+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:13.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:13.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:14.376 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:14.376+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:15.005 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:15.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:15.340 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:15.340+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:15.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:15.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:16.300 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:16.300+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:16.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:16.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:17.287 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:17.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:18.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:18.008+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:18.307 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:18.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:18.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:18.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:19.307 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:19.308+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:19.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:19.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:20.263 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:20.264+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:21.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:21.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:21.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:21.272+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:21.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:21.984+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:22.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:22.288+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:22.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:22.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:23.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:23.332+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:23.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:23.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:24.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:24.372+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:24.975 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:24.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:25.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:25.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:26.025 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:26.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:26.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:26.431+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:27.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:27.035+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:27.449 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:27.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:28.072 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:28.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:28.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:28.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:29.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:29.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:29.399 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:29.399+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:30.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:30.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:30.426 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:30.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:31.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:31.047+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:31.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:31.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:32.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:32.035+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:32.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:57:32.403 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:32.399+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:32.699 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:57:32.701 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:57:32.718 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5959,"stamp":"2024-01-12T00:57:31.788883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003226"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1976,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.366550+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:53:43.366550+0000","last_peered":"2024-01-12T00:53:43.366550+0000","last_clean":"2024-01-12T00:53:43.366550+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:53:43.366550+0000","last_undegraded":"2024-01-12T00:53:43.366550+0000","last_fullsized":"2024-01-12T00:53:43.366550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1709,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.663055+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:53:43.663055+0000","last_peered":"2024-01-12T00:53:43.663055+0000","last_clean":"2024-01-12T00:53:43.663055+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:53:43.663055+0000","last_undegraded":"2024-01-12T00:53:43.663055+0000","last_fullsized":"2024-01-12T00:53:43.663055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1921,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.961739+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:53:43.961739+0000","last_peered":"2024-01-12T00:53:43.961739+0000","last_clean":"2024-01-12T00:53:43.961739+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:53:43.961739+0000","last_undegraded":"2024-01-12T00:53:43.961739+0000","last_fullsized":"2024-01-12T00:53:43.961739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12867,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:44.543961+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:53:44.543961+0000","last_peered":"2024-01-12T00:53:44.543961+0000","last_clean":"2024-01-12T00:53:44.543961+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:53:44.543961+0000","last_undegraded":"2024-01-12T00:53:44.543961+0000","last_fullsized":"2024-01-12T00:53:44.543961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2140,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815444+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815444+0000","last_peered":"2024-01-12T00:53:48.815444+0000","last_clean":"2024-01-12T00:53:48.815444+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815444+0000","last_undegraded":"2024-01-12T00:53:48.815444+0000","last_fullsized":"2024-01-12T00:53:48.815444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12351,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.132240+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:53:45.132240+0000","last_peered":"2024-01-12T00:53:45.132240+0000","last_clean":"2024-01-12T00:53:45.132240+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:53:45.132240+0000","last_undegraded":"2024-01-12T00:53:45.132240+0000","last_fullsized":"2024-01-12T00:53:45.132240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7502,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.431908+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:53:45.431908+0000","last_peered":"2024-01-12T00:53:45.431908+0000","last_clean":"2024-01-12T00:53:45.431908+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:53:45.431908+0000","last_undegraded":"2024-01-12T00:53:45.431908+0000","last_fullsized":"2024-01-12T00:53:45.431908+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11779,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.720198+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:53:45.720198+0000","last_peered":"2024-01-12T00:53:45.720198+0000","last_clean":"2024-01-12T00:53:45.720198+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:53:45.720198+0000","last_undegraded":"2024-01-12T00:53:45.720198+0000","last_fullsized":"2024-01-12T00:53:45.720198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2517,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.016771+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:53:46.016771+0000","last_peered":"2024-01-12T00:53:46.016771+0000","last_clean":"2024-01-12T00:53:46.016771+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:53:46.016771+0000","last_undegraded":"2024-01-12T00:53:46.016771+0000","last_fullsized":"2024-01-12T00:53:46.016771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.312259+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:53:46.312259+0000","last_peered":"2024-01-12T00:53:46.312259+0000","last_clean":"2024-01-12T00:53:46.312259+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:53:46.312259+0000","last_undegraded":"2024-01-12T00:53:46.312259+0000","last_fullsized":"2024-01-12T00:53:46.312259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740695,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.45600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.69499999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.47599999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64300000000000002,"5min":1.028,"15min":1.028},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Jan 12 00:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.42599999999999999},"max":{"1min":0.753,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.69099999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78200000000000003,"5min":0.82299999999999995,"15min":1.008},"last":0.69599999999999995}]}]},{"osd":1,"up_from":8,"seq":34359740693,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.42699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68999999999999995,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.40400000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.59499999999999997}]},{"osd":2,"last update":"Fri Jan 12 00:56:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.38600000000000001},"max":{"1min":0.69199999999999995,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":1.071,"15min":1.071},"last":0.64900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359740696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64700000000000002,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.83899999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Fri Jan 12 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60999999999999999,"15min":0.60499999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.441},"max":{"1min":0.69499999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.52100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.73399999999999999,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.48699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:57:32.719 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.719 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.720 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.720 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.720 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.720 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.720 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.721 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.721 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.721 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:32.721 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:57:33.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:33.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:33.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:33.431+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:34.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:34.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:34.437 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:34.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:35.090 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:35.087+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:35.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:35.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:36.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:36.047+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:36.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:36.479+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:37.052 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:37.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:37.524 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:37.523+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:38.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:38.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:38.517 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:38.515+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:39.095 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:39.091+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:39.475 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:39.471+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:40.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:40.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:40.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:40.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:41.068 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:41.067+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:41.515 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:41.515+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:42.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:42.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:42.498 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:42.495+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:43.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:43.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:43.503 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:43.503+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:44.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:44.055+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:44.510 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:44.507+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:45.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:45.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:45.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:45.491+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:46.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:46.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:46.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:46.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:47.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:47.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:47.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:47.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:48.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:48.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:48.469 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:48.467+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:49.119 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:49.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:49.513 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:49.511+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:50.142 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:50.139+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:50.502 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:50.499+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:51.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:51.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:51.467 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:51.467+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:52.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:52.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:52.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:52.447+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:52.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:57:53.013 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:57:53.017 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:57:53.030 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5969,"stamp":"2024-01-12T00:57:51.794122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003081"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1976,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.366550+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:53:43.366550+0000","last_peered":"2024-01-12T00:53:43.366550+0000","last_clean":"2024-01-12T00:53:43.366550+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:53:43.366550+0000","last_undegraded":"2024-01-12T00:53:43.366550+0000","last_fullsized":"2024-01-12T00:53:43.366550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1709,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.663055+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:53:43.663055+0000","last_peered":"2024-01-12T00:53:43.663055+0000","last_clean":"2024-01-12T00:53:43.663055+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:53:43.663055+0000","last_undegraded":"2024-01-12T00:53:43.663055+0000","last_fullsized":"2024-01-12T00:53:43.663055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1921,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.961739+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:53:43.961739+0000","last_peered":"2024-01-12T00:53:43.961739+0000","last_clean":"2024-01-12T00:53:43.961739+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:53:43.961739+0000","last_undegraded":"2024-01-12T00:53:43.961739+0000","last_fullsized":"2024-01-12T00:53:43.961739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12867,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:44.543961+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:53:44.543961+0000","last_peered":"2024-01-12T00:53:44.543961+0000","last_clean":"2024-01-12T00:53:44.543961+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:53:44.543961+0000","last_undegraded":"2024-01-12T00:53:44.543961+0000","last_fullsized":"2024-01-12T00:53:44.543961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2140,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815444+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815444+0000","last_peered":"2024-01-12T00:53:48.815444+0000","last_clean":"2024-01-12T00:53:48.815444+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815444+0000","last_undegraded":"2024-01-12T00:53:48.815444+0000","last_fullsized":"2024-01-12T00:53:48.815444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12351,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.132240+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:53:45.132240+0000","last_peered":"2024-01-12T00:53:45.132240+0000","last_clean":"2024-01-12T00:53:45.132240+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:53:45.132240+0000","last_undegraded":"2024-01-12T00:53:45.132240+0000","last_fullsized":"2024-01-12T00:53:45.132240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7502,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.431908+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:53:45.431908+0000","last_peered":"2024-01-12T00:53:45.431908+0000","last_clean":"2024-01-12T00:53:45.431908+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:53:45.431908+0000","last_undegraded":"2024-01-12T00:53:45.431908+0000","last_fullsized":"2024-01-12T00:53:45.431908+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11779,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.720198+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:53:45.720198+0000","last_peered":"2024-01-12T00:53:45.720198+0000","last_clean":"2024-01-12T00:53:45.720198+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:53:45.720198+0000","last_undegraded":"2024-01-12T00:53:45.720198+0000","last_fullsized":"2024-01-12T00:53:45.720198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2517,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.016771+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:53:46.016771+0000","last_peered":"2024-01-12T00:53:46.016771+0000","last_clean":"2024-01-12T00:53:46.016771+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:53:46.016771+0000","last_undegraded":"2024-01-12T00:53:46.016771+0000","last_fullsized":"2024-01-12T00:53:46.016771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.312259+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:53:46.312259+0000","last_peered":"2024-01-12T00:53:46.312259+0000","last_clean":"2024-01-12T00:53:46.312259+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:53:46.312259+0000","last_undegraded":"2024-01-12T00:53:46.312259+0000","last_fullsized":"2024-01-12T00:53:46.312259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740699,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.59799999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.46899999999999997,"5min":0.39800000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":1.028,"15min":1.028},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Jan 12 00:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.76400000000000001,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.57799999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":1.008},"last":0.62}]}]},{"osd":1,"up_from":8,"seq":34359740697,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.42699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68999999999999995,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.40400000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.81100000000000005},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Jan 12 00:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73199999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.58099999999999996,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0209999999999999,"5min":1.071,"15min":1.071},"last":0.58099999999999996}]}]},{"osd":0,"up_from":8,"seq":34359740700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64700000000000002,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.83899999999999997},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60999999999999999,"15min":0.60499999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.441},"max":{"1min":0.69499999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.52100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.73399999999999999,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:57:53.031 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.032 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.032 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.032 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.032 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.032 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.033 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.033 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.033 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.033 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:57:53.033 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:57:53.122 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:53.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:53.445 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:53.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:54.131 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:54.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:54.439 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:54.439+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:55.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:55.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:55.419 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:55.419+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:56.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:56.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:56.436 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:56.435+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:57.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:57.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:57.405 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:57.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:58.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:58.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:58.446 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:58.443+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:59.100 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:57:59.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:57:59.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:57:59.423+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:00.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:00.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:00.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:00.403+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:01.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:01.019+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:01.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:01.391+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:01.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:01.975+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:02.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:02.387+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:02.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:02.939+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:03.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:03.343+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:03.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:03.911+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:04.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:04.327+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:04.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:04.955+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:05.345 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:05.343+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:05.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:05.971+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:06.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:06.351+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:07.023 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:07.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:07.333 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:07.331+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:08.069 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:08.067+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:08.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:08.371+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:09.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:09.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:09.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:09.351+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:10.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:10.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:10.323 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:10.323+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:11.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:11.043+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:11.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:11.339+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:12.024 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:12.023+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:12.378 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:12.375+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:12.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:12.987+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:13.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:58:13.328 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:58:13.331 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:58:13.341 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5979,"stamp":"2024-01-12T00:58:11.799586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003262"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1976,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.366550+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:53:43.366550+0000","last_peered":"2024-01-12T00:53:43.366550+0000","last_clean":"2024-01-12T00:53:43.366550+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:53:43.366550+0000","last_undegraded":"2024-01-12T00:53:43.366550+0000","last_fullsized":"2024-01-12T00:53:43.366550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1709,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.663055+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:53:43.663055+0000","last_peered":"2024-01-12T00:53:43.663055+0000","last_clean":"2024-01-12T00:53:43.663055+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:53:43.663055+0000","last_undegraded":"2024-01-12T00:53:43.663055+0000","last_fullsized":"2024-01-12T00:53:43.663055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1921,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:43.961739+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:53:43.961739+0000","last_peered":"2024-01-12T00:53:43.961739+0000","last_clean":"2024-01-12T00:53:43.961739+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:53:43.961739+0000","last_undegraded":"2024-01-12T00:53:43.961739+0000","last_fullsized":"2024-01-12T00:53:43.961739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12867,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:44.543961+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:53:44.543961+0000","last_peered":"2024-01-12T00:53:44.543961+0000","last_clean":"2024-01-12T00:53:44.543961+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:53:44.543961+0000","last_undegraded":"2024-01-12T00:53:44.543961+0000","last_fullsized":"2024-01-12T00:53:44.543961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2140,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:53:48.815444+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:53:48.815444+0000","last_peered":"2024-01-12T00:53:48.815444+0000","last_clean":"2024-01-12T00:53:48.815444+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:53:48.815444+0000","last_undegraded":"2024-01-12T00:53:48.815444+0000","last_fullsized":"2024-01-12T00:53:48.815444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12351,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.132240+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:53:45.132240+0000","last_peered":"2024-01-12T00:53:45.132240+0000","last_clean":"2024-01-12T00:53:45.132240+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:53:45.132240+0000","last_undegraded":"2024-01-12T00:53:45.132240+0000","last_fullsized":"2024-01-12T00:53:45.132240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7502,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.431908+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:53:45.431908+0000","last_peered":"2024-01-12T00:53:45.431908+0000","last_clean":"2024-01-12T00:53:45.431908+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:53:45.431908+0000","last_undegraded":"2024-01-12T00:53:45.431908+0000","last_fullsized":"2024-01-12T00:53:45.431908+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11779,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.720198+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:53:45.720198+0000","last_peered":"2024-01-12T00:53:45.720198+0000","last_clean":"2024-01-12T00:53:45.720198+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:53:45.720198+0000","last_undegraded":"2024-01-12T00:53:45.720198+0000","last_fullsized":"2024-01-12T00:53:45.720198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2517,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.016771+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:53:46.016771+0000","last_peered":"2024-01-12T00:53:46.016771+0000","last_clean":"2024-01-12T00:53:46.016771+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:53:46.016771+0000","last_undegraded":"2024-01-12T00:53:46.016771+0000","last_fullsized":"2024-01-12T00:53:46.016771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:46.312259+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:53:46.312259+0000","last_peered":"2024-01-12T00:53:46.312259+0000","last_clean":"2024-01-12T00:53:46.312259+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:53:46.312259+0000","last_undegraded":"2024-01-12T00:53:46.312259+0000","last_fullsized":"2024-01-12T00:53:46.312259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740703,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.59799999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.504},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.46899999999999997,"5min":0.39800000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":1.028,"15min":1.028},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Jan 12 00:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.76400000000000001,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.57799999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":1.008},"last":0.68700000000000006}]}]},{"osd":1,"up_from":8,"seq":34359740701,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Jan 12 00:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73199999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.58099999999999996,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0209999999999999,"5min":1.071,"15min":1.071},"last":0.68000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64700000000000002,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.83899999999999997},"last":0.94799999999999995}]},{"osd":2,"last update":"Fri Jan 12 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60999999999999999,"15min":0.60499999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.441},"max":{"1min":0.69499999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.52100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.73399999999999999,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:58:13.343 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.343 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.343 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.343 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.343 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.344 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.344 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.344 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.344 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.344 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:13.345 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2024-01-12T00:58:13.362 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:13.359+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:13.629 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2024-01-12T00:58:13.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2024-01-12T00:58:13.928 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2024-01-12T00:58:13.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2024-01-12T00:58:14.013 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:14.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:14.221 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2024-01-12T00:58:14.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2024-01-12T00:58:14.315 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:14.315+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:14.515 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2024-01-12T00:58:14.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2024-01-12T00:58:14.814 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2024-01-12T00:58:14.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2024-01-12T00:58:15.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:15.031+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:15.102 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-01-12T00:58:15.112 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2024-01-12T00:58:15.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:15.283+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:15.391 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2024-01-12T00:58:15.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2024-01-12T00:58:15.679 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2024-01-12T00:58:15.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2024-01-12T00:58:15.973 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2024-01-12T00:58:15.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2024-01-12T00:58:16.054 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:16.051+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:16.282 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-01-12T00:58:16.292 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:58:16.317 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:16.315+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:17.061 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:17.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:17.295 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:17.291+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:18.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:18.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:18.257 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:18.254+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:19.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:19.010+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:19.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:19.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:20.026 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:20.022+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:20.240 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:20.238+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:20.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:20.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:21.276 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:21.274+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:22.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:22.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:22.264 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:22.262+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:22.981 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:22.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:23.270 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:23.266+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:23.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:23.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:24.291 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:24.286+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:24.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:24.922+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:25.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:25.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:25.940 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:25.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:26.281 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:26.278+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:26.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:26.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:27.289 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:27.286+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:27.957 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:27.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:28.250 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:28.246+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:28.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:28.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:29.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:29.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:29.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:29.902+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:30.234 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:30.230+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:30.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:30.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:31.246 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:31.242+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:31.870 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:31.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:32.213 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:32.210+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:32.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:32.890+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:33.229 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:33.226+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:33.897 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:33.894+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:34.271 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:34.270+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:34.884 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:34.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:35.285 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:35.282+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:35.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:35.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:36.293 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:58:36.309 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:36.306+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:36.584 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:58:36.587 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:58:36.597 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":5991,"stamp":"2024-01-12T00:58:35.806159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1977,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:08.780991+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:56:08.780991+0000","last_peered":"2024-01-12T00:56:08.780991+0000","last_clean":"2024-01-12T00:56:08.780991+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:56:08.780991+0000","last_undegraded":"2024-01-12T00:56:08.780991+0000","last_fullsized":"2024-01-12T00:56:08.780991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1710,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.068373+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:56:09.068373+0000","last_peered":"2024-01-12T00:56:09.068373+0000","last_clean":"2024-01-12T00:56:09.068373+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:56:09.068373+0000","last_undegraded":"2024-01-12T00:56:09.068373+0000","last_fullsized":"2024-01-12T00:56:09.068373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1922,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.363494+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:56:09.363494+0000","last_peered":"2024-01-12T00:56:09.363494+0000","last_clean":"2024-01-12T00:56:09.363494+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:56:09.363494+0000","last_undegraded":"2024-01-12T00:56:09.363494+0000","last_fullsized":"2024-01-12T00:56:09.363494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12868,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.656743+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:56:09.656743+0000","last_peered":"2024-01-12T00:56:09.656743+0000","last_clean":"2024-01-12T00:56:09.656743+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:56:09.656743+0000","last_undegraded":"2024-01-12T00:56:09.656743+0000","last_fullsized":"2024-01-12T00:56:09.656743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2141,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:56:09.958514+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:56:09.958514+0000","last_peered":"2024-01-12T00:56:09.958514+0000","last_clean":"2024-01-12T00:56:09.958514+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:56:09.958514+0000","last_undegraded":"2024-01-12T00:56:09.958514+0000","last_fullsized":"2024-01-12T00:56:09.958514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12352,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.256791+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:56:10.256791+0000","last_peered":"2024-01-12T00:56:10.256791+0000","last_clean":"2024-01-12T00:56:10.256791+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:56:10.256791+0000","last_undegraded":"2024-01-12T00:56:10.256791+0000","last_fullsized":"2024-01-12T00:56:10.256791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7503,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.560662+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:56:10.560662+0000","last_peered":"2024-01-12T00:56:10.560662+0000","last_clean":"2024-01-12T00:56:10.560662+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:56:10.560662+0000","last_undegraded":"2024-01-12T00:56:10.560662+0000","last_fullsized":"2024-01-12T00:56:10.560662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11780,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.847929+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:56:10.847929+0000","last_peered":"2024-01-12T00:56:10.847929+0000","last_clean":"2024-01-12T00:56:10.847929+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:56:10.847929+0000","last_undegraded":"2024-01-12T00:56:10.847929+0000","last_fullsized":"2024-01-12T00:56:10.847929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2518,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.147643+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:56:11.147643+0000","last_peered":"2024-01-12T00:56:11.147643+0000","last_clean":"2024-01-12T00:56:11.147643+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:56:11.147643+0000","last_undegraded":"2024-01-12T00:56:11.147643+0000","last_fullsized":"2024-01-12T00:56:11.147643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.448942+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:56:11.448942+0000","last_peered":"2024-01-12T00:56:11.448942+0000","last_clean":"2024-01-12T00:56:11.448942+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:56:11.448942+0000","last_undegraded":"2024-01-12T00:56:11.448942+0000","last_fullsized":"2024-01-12T00:56:11.448942+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740708,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.59799999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.46899999999999997,"5min":0.39800000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":1.028,"15min":1.028},"last":0.58499999999999996}]},{"osd":1,"last update":"Fri Jan 12 00:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.76400000000000001,"5min":0.79000000000000004,"15min":0.81999999999999995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.57799999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":1.008},"last":0.68000000000000005}]}]},{"osd":1,"up_from":8,"seq":34359740706,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Jan 12 00:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73199999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.58099999999999996,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0209999999999999,"5min":1.071,"15min":1.071},"last":0.67000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359740709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.748,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.58699999999999997},"min":{"1min":0.50600000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.81599999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.48699999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:58:36.598 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.599 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.599 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.599 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.599 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.599 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.600 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.600 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.600 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.600 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:36.600 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:58:36.885 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:36.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:37.302 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:37.298+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:37.893 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:37.890+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:38.343 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:38.342+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:38.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:38.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:39.381 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:39.378+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:39.886 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:39.882+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:40.351 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:40.350+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:40.901 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:40.898+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:41.344 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:41.342+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:41.947 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:41.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:42.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:42.346+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:42.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:42.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:43.361 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:43.358+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:44.020 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:44.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:44.369 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:44.366+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:45.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:45.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:45.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:45.406+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:46.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:46.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:46.424 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:46.422+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:47.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:47.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:47.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:47.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:48.019 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:48.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:48.401 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:48.398+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:48.976 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:48.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:49.413 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:49.410+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:50.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:50.010+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:50.463 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:50.462+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:50.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:50.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:51.461 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:51.458+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:51.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:51.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:52.415 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:52.414+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:52.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:52.906+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:53.390 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:53.386+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:53.930 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:53.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:54.398 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:54.394+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:54.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:54.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:55.356 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:55.354+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:55.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:55.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:56.359 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:56.358+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:56.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:58:56.895 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:58:56.899 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:58:56.908 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6001,"stamp":"2024-01-12T00:58:55.811572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003180"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1977,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:08.780991+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:56:08.780991+0000","last_peered":"2024-01-12T00:56:08.780991+0000","last_clean":"2024-01-12T00:56:08.780991+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:56:08.780991+0000","last_undegraded":"2024-01-12T00:56:08.780991+0000","last_fullsized":"2024-01-12T00:56:08.780991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1710,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.068373+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:56:09.068373+0000","last_peered":"2024-01-12T00:56:09.068373+0000","last_clean":"2024-01-12T00:56:09.068373+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:56:09.068373+0000","last_undegraded":"2024-01-12T00:56:09.068373+0000","last_fullsized":"2024-01-12T00:56:09.068373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1922,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.363494+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:56:09.363494+0000","last_peered":"2024-01-12T00:56:09.363494+0000","last_clean":"2024-01-12T00:56:09.363494+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:56:09.363494+0000","last_undegraded":"2024-01-12T00:56:09.363494+0000","last_fullsized":"2024-01-12T00:56:09.363494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12868,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.656743+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:56:09.656743+0000","last_peered":"2024-01-12T00:56:09.656743+0000","last_clean":"2024-01-12T00:56:09.656743+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:56:09.656743+0000","last_undegraded":"2024-01-12T00:56:09.656743+0000","last_fullsized":"2024-01-12T00:56:09.656743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2141,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:56:09.958514+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:56:09.958514+0000","last_peered":"2024-01-12T00:56:09.958514+0000","last_clean":"2024-01-12T00:56:09.958514+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:56:09.958514+0000","last_undegraded":"2024-01-12T00:56:09.958514+0000","last_fullsized":"2024-01-12T00:56:09.958514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12352,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.256791+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:56:10.256791+0000","last_peered":"2024-01-12T00:56:10.256791+0000","last_clean":"2024-01-12T00:56:10.256791+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:56:10.256791+0000","last_undegraded":"2024-01-12T00:56:10.256791+0000","last_fullsized":"2024-01-12T00:56:10.256791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7503,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.560662+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:56:10.560662+0000","last_peered":"2024-01-12T00:56:10.560662+0000","last_clean":"2024-01-12T00:56:10.560662+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:56:10.560662+0000","last_undegraded":"2024-01-12T00:56:10.560662+0000","last_fullsized":"2024-01-12T00:56:10.560662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11780,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.847929+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:56:10.847929+0000","last_peered":"2024-01-12T00:56:10.847929+0000","last_clean":"2024-01-12T00:56:10.847929+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:56:10.847929+0000","last_undegraded":"2024-01-12T00:56:10.847929+0000","last_fullsized":"2024-01-12T00:56:10.847929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2518,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.147643+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:56:11.147643+0000","last_peered":"2024-01-12T00:56:11.147643+0000","last_clean":"2024-01-12T00:56:11.147643+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:56:11.147643+0000","last_undegraded":"2024-01-12T00:56:11.147643+0000","last_fullsized":"2024-01-12T00:56:11.147643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.448942+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:56:11.448942+0000","last_peered":"2024-01-12T00:56:11.448942+0000","last_clean":"2024-01-12T00:56:11.448942+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:56:11.448942+0000","last_undegraded":"2024-01-12T00:56:11.448942+0000","last_fullsized":"2024-01-12T00:56:11.448942+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740712,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.60099999999999998,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.48799999999999999,"5min":0.39800000000000002,"15min":0.33100000000000002},"max":{"1min":0.65800000000000003,"5min":0.71599999999999997,"15min":1.028},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Jan 12 00:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.81999999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78500000000000003,"5min":0.84099999999999997,"15min":1.008},"last":0.71599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740710,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.78500000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.68799999999999994}]},{"osd":2,"last update":"Fri Jan 12 00:58:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.46600000000000003,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.75600000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.56899999999999995,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72099999999999997,"5min":1.071,"15min":1.071},"last":0.754}]}]},{"osd":0,"up_from":8,"seq":34359740713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.748,"5min":0.79000000000000004,"15min":0.79200000000000004},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.58699999999999997},"min":{"1min":0.50600000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.502}]},{"osd":2,"last update":"Fri Jan 12 00:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.81599999999999995},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.48699999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:58:56.910 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.910 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.910 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.910 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.911 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.911 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.911 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.911 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.911 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.911 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:58:56.912 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:58:56.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:56.974+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:57.366 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:57.362+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:58.018 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:58.014+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:58.347 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:58.342+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:58.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:58:58.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:58:59.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:58:59.390+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:00.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:00.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:00.423 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:00.418+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:01.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:01.034+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:01.432 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:01.430+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:01.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:01.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:02.482 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:02.478+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:03.036 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:03.034+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:03.446 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:03.442+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:04.057 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:04.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:04.492 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:04.490+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:05.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:05.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:05.495 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:05.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:06.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:06.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:06.539 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:06.538+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:07.064 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:07.062+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:07.496 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:07.494+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:08.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:08.106+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:08.514 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:08.510+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:09.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:09.070+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:09.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:09.486+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:10.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:10.110+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:10.497 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:10.493+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:11.162 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:11.157+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:11.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:11.449+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:12.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:12.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:12.493 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:12.489+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:13.235 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:13.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:13.502 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:13.497+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:14.287 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:14.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:14.456 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:14.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:15.283 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:15.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:15.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:15.485+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:16.306 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:16.301+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:16.458 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:16.453+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:16.912 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:59:17.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-12T00:59:17.229+0000 7fb4c6d9d700 0 --2- 172.21.15.178:0/1742915221 >> v2:172.21.15.178:3300/0 conn(0x7fb4c808e240 0x7fb4c808cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-01-12T00:59:17.289 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:17.285+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:17.416 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:59:17.417 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:17.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:17.419 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:59:17.428 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6011,"stamp":"2024-01-12T00:59:15.816906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003141"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1977,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:08.780991+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:56:08.780991+0000","last_peered":"2024-01-12T00:56:08.780991+0000","last_clean":"2024-01-12T00:56:08.780991+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:56:08.780991+0000","last_undegraded":"2024-01-12T00:56:08.780991+0000","last_fullsized":"2024-01-12T00:56:08.780991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1710,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.068373+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:56:09.068373+0000","last_peered":"2024-01-12T00:56:09.068373+0000","last_clean":"2024-01-12T00:56:09.068373+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:56:09.068373+0000","last_undegraded":"2024-01-12T00:56:09.068373+0000","last_fullsized":"2024-01-12T00:56:09.068373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1922,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.363494+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:56:09.363494+0000","last_peered":"2024-01-12T00:56:09.363494+0000","last_clean":"2024-01-12T00:56:09.363494+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:56:09.363494+0000","last_undegraded":"2024-01-12T00:56:09.363494+0000","last_fullsized":"2024-01-12T00:56:09.363494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12868,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.656743+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:56:09.656743+0000","last_peered":"2024-01-12T00:56:09.656743+0000","last_clean":"2024-01-12T00:56:09.656743+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:56:09.656743+0000","last_undegraded":"2024-01-12T00:56:09.656743+0000","last_fullsized":"2024-01-12T00:56:09.656743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2141,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:56:09.958514+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:56:09.958514+0000","last_peered":"2024-01-12T00:56:09.958514+0000","last_clean":"2024-01-12T00:56:09.958514+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:56:09.958514+0000","last_undegraded":"2024-01-12T00:56:09.958514+0000","last_fullsized":"2024-01-12T00:56:09.958514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12352,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.256791+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:56:10.256791+0000","last_peered":"2024-01-12T00:56:10.256791+0000","last_clean":"2024-01-12T00:56:10.256791+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:56:10.256791+0000","last_undegraded":"2024-01-12T00:56:10.256791+0000","last_fullsized":"2024-01-12T00:56:10.256791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7503,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.560662+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:56:10.560662+0000","last_peered":"2024-01-12T00:56:10.560662+0000","last_clean":"2024-01-12T00:56:10.560662+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:56:10.560662+0000","last_undegraded":"2024-01-12T00:56:10.560662+0000","last_fullsized":"2024-01-12T00:56:10.560662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11780,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.847929+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:56:10.847929+0000","last_peered":"2024-01-12T00:56:10.847929+0000","last_clean":"2024-01-12T00:56:10.847929+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:56:10.847929+0000","last_undegraded":"2024-01-12T00:56:10.847929+0000","last_fullsized":"2024-01-12T00:56:10.847929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2518,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.147643+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:56:11.147643+0000","last_peered":"2024-01-12T00:56:11.147643+0000","last_clean":"2024-01-12T00:56:11.147643+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:56:11.147643+0000","last_undegraded":"2024-01-12T00:56:11.147643+0000","last_fullsized":"2024-01-12T00:56:11.147643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.448942+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:56:11.448942+0000","last_peered":"2024-01-12T00:56:11.448942+0000","last_clean":"2024-01-12T00:56:11.448942+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:56:11.448942+0000","last_undegraded":"2024-01-12T00:56:11.448942+0000","last_fullsized":"2024-01-12T00:56:11.448942+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740716,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.60099999999999998,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.443},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.48799999999999999,"5min":0.39800000000000002,"15min":0.33100000000000002},"max":{"1min":0.65800000000000003,"5min":0.71599999999999997,"15min":1.028},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Jan 12 00:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.81999999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78500000000000003,"5min":0.84099999999999997,"15min":1.008},"last":0.64200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740714,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.42099999999999999,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.60499999999999998,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.623,"15min":0.621},"min":{"1min":0.54000000000000004,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.70699999999999996,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Jan 12 00:58:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.46600000000000003,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.75600000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.56899999999999995,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72099999999999997,"5min":1.071,"15min":1.071},"last":0.64700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359740717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.42499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.79000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 12 00:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.81599999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.72299999999999998,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.58699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:59:17.430 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.430 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.430 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.430 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.430 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.431 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.431 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.431 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.431 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.431 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:17.431 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:59:18.269 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:18.265+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:18.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:18.381+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:19.242 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:19.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:19.352 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:19.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:20.205 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:20.201+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:20.363 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:20.361+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:21.227 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:21.221+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:21.321 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:21.317+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:22.238 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:22.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:22.327 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:22.321+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:23.193 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:23.189+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:23.341 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:23.337+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:24.237 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:24.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:24.306 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:24.301+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:25.203 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:25.197+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:25.338 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:25.333+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:26.241 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:26.237+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:26.331 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:26.329+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:27.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:27.209+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:27.360 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:27.357+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:28.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:28.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:28.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:28.325+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:29.244 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:29.241+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:29.312 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:29.309+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:30.236 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:30.233+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:30.353 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:30.349+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:31.260 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:31.257+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:31.368 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:31.365+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:32.294 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:32.289+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:32.405 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:32.401+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:33.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:33.261+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:33.416 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:33.413+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:34.254 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:34.249+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:34.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:34.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:35.298 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:35.293+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:35.444 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:35.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:36.326 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:36.321+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:36.481 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:36.477+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:37.332 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:37.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:37.432 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:59:37.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:37.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:37.723 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:59:37.726 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:59:37.736 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6021,"stamp":"2024-01-12T00:59:35.822448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1977,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:08.780991+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:56:08.780991+0000","last_peered":"2024-01-12T00:56:08.780991+0000","last_clean":"2024-01-12T00:56:08.780991+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:56:08.780991+0000","last_undegraded":"2024-01-12T00:56:08.780991+0000","last_fullsized":"2024-01-12T00:56:08.780991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1710,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.068373+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:56:09.068373+0000","last_peered":"2024-01-12T00:56:09.068373+0000","last_clean":"2024-01-12T00:56:09.068373+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:56:09.068373+0000","last_undegraded":"2024-01-12T00:56:09.068373+0000","last_fullsized":"2024-01-12T00:56:09.068373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1922,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.363494+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:56:09.363494+0000","last_peered":"2024-01-12T00:56:09.363494+0000","last_clean":"2024-01-12T00:56:09.363494+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:56:09.363494+0000","last_undegraded":"2024-01-12T00:56:09.363494+0000","last_fullsized":"2024-01-12T00:56:09.363494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12868,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.656743+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:56:09.656743+0000","last_peered":"2024-01-12T00:56:09.656743+0000","last_clean":"2024-01-12T00:56:09.656743+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:56:09.656743+0000","last_undegraded":"2024-01-12T00:56:09.656743+0000","last_fullsized":"2024-01-12T00:56:09.656743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2141,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:56:09.958514+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:56:09.958514+0000","last_peered":"2024-01-12T00:56:09.958514+0000","last_clean":"2024-01-12T00:56:09.958514+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:56:09.958514+0000","last_undegraded":"2024-01-12T00:56:09.958514+0000","last_fullsized":"2024-01-12T00:56:09.958514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12352,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.256791+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:56:10.256791+0000","last_peered":"2024-01-12T00:56:10.256791+0000","last_clean":"2024-01-12T00:56:10.256791+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:56:10.256791+0000","last_undegraded":"2024-01-12T00:56:10.256791+0000","last_fullsized":"2024-01-12T00:56:10.256791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7503,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.560662+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:56:10.560662+0000","last_peered":"2024-01-12T00:56:10.560662+0000","last_clean":"2024-01-12T00:56:10.560662+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:56:10.560662+0000","last_undegraded":"2024-01-12T00:56:10.560662+0000","last_fullsized":"2024-01-12T00:56:10.560662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11780,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.847929+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:56:10.847929+0000","last_peered":"2024-01-12T00:56:10.847929+0000","last_clean":"2024-01-12T00:56:10.847929+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:56:10.847929+0000","last_undegraded":"2024-01-12T00:56:10.847929+0000","last_fullsized":"2024-01-12T00:56:10.847929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2518,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.147643+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:56:11.147643+0000","last_peered":"2024-01-12T00:56:11.147643+0000","last_clean":"2024-01-12T00:56:11.147643+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:56:11.147643+0000","last_undegraded":"2024-01-12T00:56:11.147643+0000","last_fullsized":"2024-01-12T00:56:11.147643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.448942+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:56:11.448942+0000","last_peered":"2024-01-12T00:56:11.448942+0000","last_clean":"2024-01-12T00:56:11.448942+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:56:11.448942+0000","last_undegraded":"2024-01-12T00:56:11.448942+0000","last_fullsized":"2024-01-12T00:56:11.448942+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740720,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.60099999999999998,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.48799999999999999,"5min":0.39800000000000002,"15min":0.33100000000000002},"max":{"1min":0.65800000000000003,"5min":0.71599999999999997,"15min":1.028},"last":0.56899999999999995}]},{"osd":1,"last update":"Fri Jan 12 00:58:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.81999999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78500000000000003,"5min":0.84099999999999997,"15min":1.008},"last":0.66400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359740718,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.42099999999999999,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.60499999999999998,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.623,"15min":0.621},"min":{"1min":0.54000000000000004,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.70699999999999996,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Jan 12 00:58:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.46600000000000003,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.75600000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.56899999999999995,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72099999999999997,"5min":1.071,"15min":1.071},"last":0.71599999999999997}]}]},{"osd":0,"up_from":8,"seq":34359740721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.42499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.79000000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.63700000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.81599999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.72299999999999998,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:59:37.737 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.737 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.737 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.738 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.738 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.738 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.738 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.738 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.738 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.739 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:37.739 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:59:38.284 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:38.281+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:38.448 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:38.445+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:39.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:39.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:39.430 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:39.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:40.282 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:40.277+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:40.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:40.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:41.316 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:41.313+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:41.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:41.389+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:42.363 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:42.357+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:42.382 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:42.377+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:43.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:43.341+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:43.388 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:43.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:44.390 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:44.385+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:44.440 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:44.437+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:45.374 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:45.369+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:45.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:45.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:46.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:46.373+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:46.400 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:46.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:47.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:47.389+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:47.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:47.421+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:48.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:48.377+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:48.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:48.417+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:49.372 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:49.369+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:49.381 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:49.377+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:50.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:50.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:50.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:50.397+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:51.356 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:51.353+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:51.411 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:51.409+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:52.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:52.333+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:52.387 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:52.385+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:53.389 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:53.385+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:53.409 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:53.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:54.384 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:54.381+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:54.443 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:54.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:55.347 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:55.345+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:55.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:55.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:56.333 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:56.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:56.408 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:56.405+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:57.346 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:57.341+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:57.433 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:57.429+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:57.739 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T00:59:58.033 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T00:59:58.036 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T00:59:58.046 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6032,"stamp":"2024-01-12T00:59:57.828286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003245"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1977,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:08.780991+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:56:08.780991+0000","last_peered":"2024-01-12T00:56:08.780991+0000","last_clean":"2024-01-12T00:56:08.780991+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:56:08.780991+0000","last_undegraded":"2024-01-12T00:56:08.780991+0000","last_fullsized":"2024-01-12T00:56:08.780991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1710,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.068373+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:56:09.068373+0000","last_peered":"2024-01-12T00:56:09.068373+0000","last_clean":"2024-01-12T00:56:09.068373+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:56:09.068373+0000","last_undegraded":"2024-01-12T00:56:09.068373+0000","last_fullsized":"2024-01-12T00:56:09.068373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1922,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.363494+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:56:09.363494+0000","last_peered":"2024-01-12T00:56:09.363494+0000","last_clean":"2024-01-12T00:56:09.363494+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:56:09.363494+0000","last_undegraded":"2024-01-12T00:56:09.363494+0000","last_fullsized":"2024-01-12T00:56:09.363494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12868,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.656743+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:56:09.656743+0000","last_peered":"2024-01-12T00:56:09.656743+0000","last_clean":"2024-01-12T00:56:09.656743+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:56:09.656743+0000","last_undegraded":"2024-01-12T00:56:09.656743+0000","last_fullsized":"2024-01-12T00:56:09.656743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2141,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:56:09.958514+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:56:09.958514+0000","last_peered":"2024-01-12T00:56:09.958514+0000","last_clean":"2024-01-12T00:56:09.958514+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:56:09.958514+0000","last_undegraded":"2024-01-12T00:56:09.958514+0000","last_fullsized":"2024-01-12T00:56:09.958514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12352,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.256791+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:56:10.256791+0000","last_peered":"2024-01-12T00:56:10.256791+0000","last_clean":"2024-01-12T00:56:10.256791+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:56:10.256791+0000","last_undegraded":"2024-01-12T00:56:10.256791+0000","last_fullsized":"2024-01-12T00:56:10.256791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7503,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.560662+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:56:10.560662+0000","last_peered":"2024-01-12T00:56:10.560662+0000","last_clean":"2024-01-12T00:56:10.560662+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:56:10.560662+0000","last_undegraded":"2024-01-12T00:56:10.560662+0000","last_fullsized":"2024-01-12T00:56:10.560662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11780,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.847929+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:56:10.847929+0000","last_peered":"2024-01-12T00:56:10.847929+0000","last_clean":"2024-01-12T00:56:10.847929+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:56:10.847929+0000","last_undegraded":"2024-01-12T00:56:10.847929+0000","last_fullsized":"2024-01-12T00:56:10.847929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2518,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.147643+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:56:11.147643+0000","last_peered":"2024-01-12T00:56:11.147643+0000","last_clean":"2024-01-12T00:56:11.147643+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:56:11.147643+0000","last_undegraded":"2024-01-12T00:56:11.147643+0000","last_fullsized":"2024-01-12T00:56:11.147643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.448942+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:56:11.448942+0000","last_peered":"2024-01-12T00:56:11.448942+0000","last_clean":"2024-01-12T00:56:11.448942+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:56:11.448942+0000","last_undegraded":"2024-01-12T00:56:11.448942+0000","last_fullsized":"2024-01-12T00:56:11.448942+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740724,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:59:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52300000000000002,"15min":0.52800000000000002},"min":{"1min":0.42399999999999999,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.68999999999999995,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.33100000000000002},"max":{"1min":0.68799999999999994,"5min":0.71599999999999997,"15min":1.028},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Jan 12 00:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.81999999999999995},"last":0.505},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68000000000000005,"15min":0.68999999999999995},"min":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":1.008},"last":0.67400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740722,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.42099999999999999,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.60499999999999998,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.499},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.623,"15min":0.621},"min":{"1min":0.54000000000000004,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.70699999999999996,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.55900000000000005,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76700000000000002,"5min":1.0209999999999999,"15min":1.071},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359740725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 00:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.42499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73699999999999999,"5min":0.78400000000000003,"15min":0.79000000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.63400000000000001}]},{"osd":2,"last update":"Fri Jan 12 00:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.81599999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.72299999999999998,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.58399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T00:59:58.048 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.048 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.048 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.048 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.049 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.049 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.049 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.049 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.049 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.049 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T00:59:58.050 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T00:59:58.339 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:58.337+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:58.462 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:58.457+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:59.319 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T00:59:59.317+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T00:59:59.471 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T00:59:59.469+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:00.308 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:00.305+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:00.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:00.425+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:01.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:01.329+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:01.447 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:01.441+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:02.371 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:02.365+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:02.402 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:02.397+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:03.358 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:03.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:03.453 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:03.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:04.369 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:04.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:04.442 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:04.436+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:05.393 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:05.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:05.457 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:05.452+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:06.405 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:06.400+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:06.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:06.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:07.399 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:07.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:07.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:07.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:08.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:08.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:08.454 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:08.448+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:09.338 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:09.332+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:09.434 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:09.428+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:10.378 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:10.372+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:10.406 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:10.400+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:11.352 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:11.348+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:11.371 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:11.368+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:12.332 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:12.328+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:12.348 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:12.344+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:13.346 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:13.340+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:13.357 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:13.352+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:14.311 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:14.304+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:14.397 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:14.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:15.339 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:15.336+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:15.428 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:15.424+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:16.386 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:16.380+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:16.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:16.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:17.414 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:17.408+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:17.493 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:17.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:18.051 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:00:18.345 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:00:18.348 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:00:18.357 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6042,"stamp":"2024-01-12T01:00:17.833692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003266"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1977,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:08.780991+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:56:08.780991+0000","last_peered":"2024-01-12T00:56:08.780991+0000","last_clean":"2024-01-12T00:56:08.780991+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:56:08.780991+0000","last_undegraded":"2024-01-12T00:56:08.780991+0000","last_fullsized":"2024-01-12T00:56:08.780991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1710,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.068373+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:56:09.068373+0000","last_peered":"2024-01-12T00:56:09.068373+0000","last_clean":"2024-01-12T00:56:09.068373+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:56:09.068373+0000","last_undegraded":"2024-01-12T00:56:09.068373+0000","last_fullsized":"2024-01-12T00:56:09.068373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1922,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.363494+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:56:09.363494+0000","last_peered":"2024-01-12T00:56:09.363494+0000","last_clean":"2024-01-12T00:56:09.363494+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:56:09.363494+0000","last_undegraded":"2024-01-12T00:56:09.363494+0000","last_fullsized":"2024-01-12T00:56:09.363494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12868,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:09.656743+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:56:09.656743+0000","last_peered":"2024-01-12T00:56:09.656743+0000","last_clean":"2024-01-12T00:56:09.656743+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:56:09.656743+0000","last_undegraded":"2024-01-12T00:56:09.656743+0000","last_fullsized":"2024-01-12T00:56:09.656743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2141,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:56:09.958514+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:56:09.958514+0000","last_peered":"2024-01-12T00:56:09.958514+0000","last_clean":"2024-01-12T00:56:09.958514+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:56:09.958514+0000","last_undegraded":"2024-01-12T00:56:09.958514+0000","last_fullsized":"2024-01-12T00:56:09.958514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12352,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.256791+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:56:10.256791+0000","last_peered":"2024-01-12T00:56:10.256791+0000","last_clean":"2024-01-12T00:56:10.256791+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:56:10.256791+0000","last_undegraded":"2024-01-12T00:56:10.256791+0000","last_fullsized":"2024-01-12T00:56:10.256791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7503,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.560662+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:56:10.560662+0000","last_peered":"2024-01-12T00:56:10.560662+0000","last_clean":"2024-01-12T00:56:10.560662+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:56:10.560662+0000","last_undegraded":"2024-01-12T00:56:10.560662+0000","last_fullsized":"2024-01-12T00:56:10.560662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11780,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:10.847929+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:56:10.847929+0000","last_peered":"2024-01-12T00:56:10.847929+0000","last_clean":"2024-01-12T00:56:10.847929+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:56:10.847929+0000","last_undegraded":"2024-01-12T00:56:10.847929+0000","last_fullsized":"2024-01-12T00:56:10.847929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2518,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.147643+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:56:11.147643+0000","last_peered":"2024-01-12T00:56:11.147643+0000","last_clean":"2024-01-12T00:56:11.147643+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:56:11.147643+0000","last_undegraded":"2024-01-12T00:56:11.147643+0000","last_fullsized":"2024-01-12T00:56:11.147643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:56:11.448942+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:56:11.448942+0000","last_peered":"2024-01-12T00:56:11.448942+0000","last_clean":"2024-01-12T00:56:11.448942+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:56:11.448942+0000","last_undegraded":"2024-01-12T00:56:11.448942+0000","last_fullsized":"2024-01-12T00:56:11.448942+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740728,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:59:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52300000000000002,"15min":0.52800000000000002},"min":{"1min":0.42399999999999999,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.68999999999999995,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.33100000000000002},"max":{"1min":0.68799999999999994,"5min":0.71599999999999997,"15min":1.028},"last":0.62}]},{"osd":1,"last update":"Fri Jan 12 00:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.81999999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68000000000000005,"15min":0.68999999999999995},"min":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":1.008},"last":0.70699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359740726,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:00:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.44,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.63,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.54600000000000004,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.78400000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Jan 12 00:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.55900000000000005,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76700000000000002,"5min":1.0209999999999999,"15min":1.071},"last":0.68200000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68700000000000006,"5min":0.748,"15min":0.79000000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.52800000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67300000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.58699999999999997}]},{"osd":2,"last update":"Fri Jan 12 01:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.60499999999999998},"min":{"1min":0.55400000000000005,"5min":0.439,"15min":0.439},"max":{"1min":0.71699999999999997,"5min":0.77000000000000002,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.53000000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.754,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:00:18.359 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.359 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.359 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.359 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.359 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.360 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.360 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.360 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.360 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.360 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:18.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2024-01-12T01:00:18.420 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:18.416+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:18.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:18.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:18.656 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2024-01-12T01:00:18.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2024-01-12T01:00:18.957 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2024-01-12T01:00:18.972 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2024-01-12T01:00:19.259 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2024-01-12T01:00:19.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2024-01-12T01:00:19.410 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:19.404+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:19.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:19.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:19.553 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2024-01-12T01:00:19.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2024-01-12T01:00:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2024-01-12T01:00:19.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2024-01-12T01:00:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-01-12T01:00:20.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2024-01-12T01:00:20.393 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:20.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:20.449 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2024-01-12T01:00:20.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2024-01-12T01:00:20.485 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:20.480+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:20.753 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2024-01-12T01:00:20.766 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2024-01-12T01:00:21.053 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2024-01-12T01:00:21.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2024-01-12T01:00:21.352 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-01-12T01:00:21.364 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:00:21.374 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:21.368+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:21.478 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:21.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:22.330 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:22.324+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:22.479 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:22.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:23.308 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:23.304+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:23.468 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:23.464+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:24.350 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:24.344+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:24.451 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:24.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:25.394 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:25.388+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:25.407 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:25.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:26.438 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:26.432+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:26.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:26.440+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:27.487 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:27.484+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:27.491 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:27.488+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:28.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:28.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:28.464 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:28.460+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:29.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:29.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:29.428 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:29.424+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:30.418 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:30.412+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:30.425 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:30.420+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:31.370 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:31.364+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:31.473 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:31.468+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:32.394 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:32.388+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:32.455 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:32.452+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:33.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:33.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:33.491 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:33.488+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:34.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:34.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:34.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:34.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:35.433 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:35.428+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:35.508 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:35.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:36.400 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:36.396+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:36.504 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:36.500+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:37.398 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:37.392+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:37.551 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:37.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:38.438 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:38.432+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:38.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:38.560+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:39.477 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:39.476+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:39.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:39.552+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:40.497 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:40.496+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:40.509 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:40.504+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:41.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:00:41.462 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:41.456+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:41.525 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:41.524+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:41.661 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:00:41.664 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:00:41.674 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6053,"stamp":"2024-01-12T01:00:39.839428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003045"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1978,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.627179+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:58:13.627179+0000","last_peered":"2024-01-12T00:58:13.627179+0000","last_clean":"2024-01-12T00:58:13.627179+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:58:13.627179+0000","last_undegraded":"2024-01-12T00:58:13.627179+0000","last_fullsized":"2024-01-12T00:58:13.627179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1711,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.924040+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:58:13.924040+0000","last_peered":"2024-01-12T00:58:13.924040+0000","last_clean":"2024-01-12T00:58:13.924040+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:58:13.924040+0000","last_undegraded":"2024-01-12T00:58:13.924040+0000","last_fullsized":"2024-01-12T00:58:13.924040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1923,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.218643+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:58:14.218643+0000","last_peered":"2024-01-12T00:58:14.218643+0000","last_clean":"2024-01-12T00:58:14.218643+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:58:14.218643+0000","last_undegraded":"2024-01-12T00:58:14.218643+0000","last_fullsized":"2024-01-12T00:58:14.218643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12869,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.512557+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:58:14.512557+0000","last_peered":"2024-01-12T00:58:14.512557+0000","last_clean":"2024-01-12T00:58:14.512557+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:58:14.512557+0000","last_undegraded":"2024-01-12T00:58:14.512557+0000","last_fullsized":"2024-01-12T00:58:14.512557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2142,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:58:14.811998+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:58:14.811998+0000","last_peered":"2024-01-12T00:58:14.811998+0000","last_clean":"2024-01-12T00:58:14.811998+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:58:14.811998+0000","last_undegraded":"2024-01-12T00:58:14.811998+0000","last_fullsized":"2024-01-12T00:58:14.811998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 391s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12353,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.100101+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:58:15.100101+0000","last_peered":"2024-01-12T00:58:15.100101+0000","last_clean":"2024-01-12T00:58:15.100101+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:58:15.100101+0000","last_undegraded":"2024-01-12T00:58:15.100101+0000","last_fullsized":"2024-01-12T00:58:15.100101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7504,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.389356+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:58:15.389356+0000","last_peered":"2024-01-12T00:58:15.389356+0000","last_clean":"2024-01-12T00:58:15.389356+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:58:15.389356+0000","last_undegraded":"2024-01-12T00:58:15.389356+0000","last_fullsized":"2024-01-12T00:58:15.389356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11781,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.676634+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:58:15.676634+0000","last_peered":"2024-01-12T00:58:15.676634+0000","last_clean":"2024-01-12T00:58:15.676634+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:58:15.676634+0000","last_undegraded":"2024-01-12T00:58:15.676634+0000","last_fullsized":"2024-01-12T00:58:15.676634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2519,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.969318+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:58:15.969318+0000","last_peered":"2024-01-12T00:58:15.969318+0000","last_clean":"2024-01-12T00:58:15.969318+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:58:15.969318+0000","last_undegraded":"2024-01-12T00:58:15.969318+0000","last_fullsized":"2024-01-12T00:58:15.969318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:16.279475+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:58:16.279475+0000","last_peered":"2024-01-12T00:58:16.279475+0000","last_clean":"2024-01-12T00:58:16.279475+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:58:16.279475+0000","last_undegraded":"2024-01-12T00:58:16.279475+0000","last_fullsized":"2024-01-12T00:58:16.279475+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740732,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 00:59:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52300000000000002,"15min":0.52800000000000002},"min":{"1min":0.42399999999999999,"5min":0.39200000000000002,"15min":0.37},"max":{"1min":0.68999999999999995,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.33100000000000002},"max":{"1min":0.68799999999999994,"5min":0.71599999999999997,"15min":1.028},"last":0.66900000000000004}]},{"osd":1,"last update":"Fri Jan 12 00:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.442,"15min":0.42599999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.81999999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68000000000000005,"15min":0.68999999999999995},"min":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":1.008},"last":0.80500000000000005}]}]},{"osd":1,"up_from":8,"seq":34359740730,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:00:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.44,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.63,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.54600000000000004,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.78400000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.76800000000000002}]},{"osd":2,"last update":"Fri Jan 12 00:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.55900000000000005,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76700000000000002,"5min":1.0209999999999999,"15min":1.071},"last":0.67900000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68700000000000006,"5min":0.748,"15min":0.79000000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.52800000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67300000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.67800000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.60499999999999998},"min":{"1min":0.55400000000000005,"5min":0.439,"15min":0.439},"max":{"1min":0.71699999999999997,"5min":0.77000000000000002,"15min":0.81599999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.53000000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.754,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:00:41.675 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.675 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.676 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.676 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.676 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.676 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.677 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.677 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.677 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.677 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:00:41.677 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:00:42.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:42.416+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:42.533 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:42.532+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:43.404 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:43.404+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:43.548 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:43.548+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:44.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:44.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:44.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:44.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:45.473 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:45.472+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:45.532 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:45.532+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:46.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:46.448+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:46.564 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:46.564+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:47.454 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:47.452+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:47.545 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:47.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:48.436 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:48.436+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:48.537 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:48.536+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:49.463 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:49.460+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:49.585 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:49.584+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:50.501 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:50.500+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:50.557 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:50.556+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:51.531 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:51.532+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:51.549 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:51.548+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:52.488 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:52.488+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:52.585 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:52.584+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:53.519 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:53.516+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:53.600 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:53.600+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:54.543 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:54.544+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:54.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:54.616+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:55.574 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:55.571+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:55.611 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:55.611+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:56.529 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:56.527+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:56.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:56.583+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:57.565 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:57.563+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:57.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:57.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:58.575 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:58.571+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:58.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:58.619+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:59.600 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:00:59.599+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:00:59.651 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:00:59.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:00.560 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:00.559+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:00.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:00.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:01.581 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:01.579+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:01.612 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:01.611+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:01.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:01:01.966 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:01:01.968 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:01:01.978 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6064,"stamp":"2024-01-12T01:01:01.845577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003472"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1978,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.627179+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:58:13.627179+0000","last_peered":"2024-01-12T00:58:13.627179+0000","last_clean":"2024-01-12T00:58:13.627179+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:58:13.627179+0000","last_undegraded":"2024-01-12T00:58:13.627179+0000","last_fullsized":"2024-01-12T00:58:13.627179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1711,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.924040+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:58:13.924040+0000","last_peered":"2024-01-12T00:58:13.924040+0000","last_clean":"2024-01-12T00:58:13.924040+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:58:13.924040+0000","last_undegraded":"2024-01-12T00:58:13.924040+0000","last_fullsized":"2024-01-12T00:58:13.924040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1923,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.218643+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:58:14.218643+0000","last_peered":"2024-01-12T00:58:14.218643+0000","last_clean":"2024-01-12T00:58:14.218643+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:58:14.218643+0000","last_undegraded":"2024-01-12T00:58:14.218643+0000","last_fullsized":"2024-01-12T00:58:14.218643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12869,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.512557+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:58:14.512557+0000","last_peered":"2024-01-12T00:58:14.512557+0000","last_clean":"2024-01-12T00:58:14.512557+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:58:14.512557+0000","last_undegraded":"2024-01-12T00:58:14.512557+0000","last_fullsized":"2024-01-12T00:58:14.512557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2142,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:58:14.811998+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:58:14.811998+0000","last_peered":"2024-01-12T00:58:14.811998+0000","last_clean":"2024-01-12T00:58:14.811998+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:58:14.811998+0000","last_undegraded":"2024-01-12T00:58:14.811998+0000","last_fullsized":"2024-01-12T00:58:14.811998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 391s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12353,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.100101+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:58:15.100101+0000","last_peered":"2024-01-12T00:58:15.100101+0000","last_clean":"2024-01-12T00:58:15.100101+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:58:15.100101+0000","last_undegraded":"2024-01-12T00:58:15.100101+0000","last_fullsized":"2024-01-12T00:58:15.100101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7504,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.389356+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:58:15.389356+0000","last_peered":"2024-01-12T00:58:15.389356+0000","last_clean":"2024-01-12T00:58:15.389356+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:58:15.389356+0000","last_undegraded":"2024-01-12T00:58:15.389356+0000","last_fullsized":"2024-01-12T00:58:15.389356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11781,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.676634+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:58:15.676634+0000","last_peered":"2024-01-12T00:58:15.676634+0000","last_clean":"2024-01-12T00:58:15.676634+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:58:15.676634+0000","last_undegraded":"2024-01-12T00:58:15.676634+0000","last_fullsized":"2024-01-12T00:58:15.676634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2519,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.969318+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:58:15.969318+0000","last_peered":"2024-01-12T00:58:15.969318+0000","last_clean":"2024-01-12T00:58:15.969318+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:58:15.969318+0000","last_undegraded":"2024-01-12T00:58:15.969318+0000","last_fullsized":"2024-01-12T00:58:15.969318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:16.279475+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:58:16.279475+0000","last_peered":"2024-01-12T00:58:16.279475+0000","last_clean":"2024-01-12T00:58:16.279475+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:58:16.279475+0000","last_undegraded":"2024-01-12T00:58:16.279475+0000","last_fullsized":"2024-01-12T00:58:16.279475+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740737,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.622,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.67800000000000005,"5min":0.70799999999999996,"15min":1.028},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Jan 12 01:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71599999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80500000000000005,"5min":0.84099999999999997,"15min":1.008},"last":0.72999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740735,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:00:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.44,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.63,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.54600000000000004,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.78400000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jan 12 01:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46400000000000002,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73399999999999999,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66600000000000004,"15min":0.66500000000000004},"min":{"1min":0.57299999999999995,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83599999999999997,"5min":1.0209999999999999,"15min":1.071},"last":0.65600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359740738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68700000000000006,"5min":0.748,"15min":0.79000000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.52800000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67300000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.60499999999999998},"min":{"1min":0.55400000000000005,"5min":0.439,"15min":0.439},"max":{"1min":0.71699999999999997,"5min":0.77000000000000002,"15min":0.81599999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.53000000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.754,"5min":0.77700000000000002,"15min":0.91600000000000004},"last":0.64700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:01:01.980 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.980 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.980 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.980 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.980 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.981 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.981 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.981 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.981 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.981 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:01.982 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:01:02.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:02.599+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:02.624 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:02.623+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:03.579 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:03.579+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:03.672 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:03.671+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:04.597 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:04.595+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:04.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:04.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:05.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:05.615+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:05.702 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:05.699+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:06.646 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:06.643+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:06.668 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:06.667+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:07.667 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:07.667+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:07.690 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:07.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:08.667 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:08.663+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:08.672 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:08.671+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:09.632 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:09.631+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:09.633 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:09.631+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:10.658 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:10.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:10.680 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:10.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:11.615 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:11.611+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:11.661 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:11.659+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:12.641 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:12.639+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:12.655 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:12.655+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:13.633 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:13.631+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:13.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:13.651+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:14.616 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:14.615+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:14.639 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:14.639+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:15.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:15.631+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:15.688 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:15.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:16.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:16.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:16.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:16.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:17.653 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:17.651+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:17.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:17.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:18.627 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:18.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:18.767 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:18.763+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:19.648 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:19.647+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:19.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:19.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:20.656 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:20.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:20.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:20.723+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:21.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:21.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:21.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:21.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:21.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:01:22.273 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:01:22.275 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:01:22.291 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6074,"stamp":"2024-01-12T01:01:21.851078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003246"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1978,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.627179+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:58:13.627179+0000","last_peered":"2024-01-12T00:58:13.627179+0000","last_clean":"2024-01-12T00:58:13.627179+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:58:13.627179+0000","last_undegraded":"2024-01-12T00:58:13.627179+0000","last_fullsized":"2024-01-12T00:58:13.627179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1711,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.924040+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:58:13.924040+0000","last_peered":"2024-01-12T00:58:13.924040+0000","last_clean":"2024-01-12T00:58:13.924040+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:58:13.924040+0000","last_undegraded":"2024-01-12T00:58:13.924040+0000","last_fullsized":"2024-01-12T00:58:13.924040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1923,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.218643+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:58:14.218643+0000","last_peered":"2024-01-12T00:58:14.218643+0000","last_clean":"2024-01-12T00:58:14.218643+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:58:14.218643+0000","last_undegraded":"2024-01-12T00:58:14.218643+0000","last_fullsized":"2024-01-12T00:58:14.218643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12869,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.512557+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:58:14.512557+0000","last_peered":"2024-01-12T00:58:14.512557+0000","last_clean":"2024-01-12T00:58:14.512557+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:58:14.512557+0000","last_undegraded":"2024-01-12T00:58:14.512557+0000","last_fullsized":"2024-01-12T00:58:14.512557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2142,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:58:14.811998+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:58:14.811998+0000","last_peered":"2024-01-12T00:58:14.811998+0000","last_clean":"2024-01-12T00:58:14.811998+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:58:14.811998+0000","last_undegraded":"2024-01-12T00:58:14.811998+0000","last_fullsized":"2024-01-12T00:58:14.811998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 391s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12353,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.100101+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:58:15.100101+0000","last_peered":"2024-01-12T00:58:15.100101+0000","last_clean":"2024-01-12T00:58:15.100101+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:58:15.100101+0000","last_undegraded":"2024-01-12T00:58:15.100101+0000","last_fullsized":"2024-01-12T00:58:15.100101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7504,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.389356+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:58:15.389356+0000","last_peered":"2024-01-12T00:58:15.389356+0000","last_clean":"2024-01-12T00:58:15.389356+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:58:15.389356+0000","last_undegraded":"2024-01-12T00:58:15.389356+0000","last_fullsized":"2024-01-12T00:58:15.389356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11781,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.676634+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:58:15.676634+0000","last_peered":"2024-01-12T00:58:15.676634+0000","last_clean":"2024-01-12T00:58:15.676634+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:58:15.676634+0000","last_undegraded":"2024-01-12T00:58:15.676634+0000","last_fullsized":"2024-01-12T00:58:15.676634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2519,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.969318+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:58:15.969318+0000","last_peered":"2024-01-12T00:58:15.969318+0000","last_clean":"2024-01-12T00:58:15.969318+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:58:15.969318+0000","last_undegraded":"2024-01-12T00:58:15.969318+0000","last_fullsized":"2024-01-12T00:58:15.969318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:16.279475+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:58:16.279475+0000","last_peered":"2024-01-12T00:58:16.279475+0000","last_clean":"2024-01-12T00:58:16.279475+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:58:16.279475+0000","last_undegraded":"2024-01-12T00:58:16.279475+0000","last_fullsized":"2024-01-12T00:58:16.279475+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740741,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.622,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.44},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.67800000000000005,"5min":0.70799999999999996,"15min":1.028},"last":0.5}]},{"osd":1,"last update":"Fri Jan 12 01:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71599999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80500000000000005,"5min":0.84099999999999997,"15min":1.008},"last":0.65800000000000003}]}]},{"osd":1,"up_from":8,"seq":34359740739,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.61799999999999999,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.51200000000000001,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.77800000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.66600000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46400000000000002,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73399999999999999,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.503},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66600000000000004,"15min":0.66500000000000004},"min":{"1min":0.57299999999999995,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83599999999999997,"5min":1.0209999999999999,"15min":1.071},"last":0.55900000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64400000000000002,"5min":0.748,"15min":0.79000000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.51300000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.754,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.48799999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.91600000000000004},"last":0.70299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:01:22.293 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.293 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.293 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.293 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.293 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.293 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.294 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.294 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.294 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.294 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:22.294 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:01:22.670 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:22.667+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:22.727 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:22.723+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:23.703 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:23.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:23.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:23.743+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:24.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:24.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:24.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:24.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:25.624 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:25.623+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:25.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:25.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:26.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:26.627+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:26.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:26.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:27.655 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:27.655+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:27.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:27.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:28.692 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:28.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:28.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:28.775+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:29.704 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:29.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:29.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:29.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:30.733 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:30.731+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:30.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:30.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:31.702 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:31.699+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:31.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:31.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:32.731 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:32.727+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:32.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:32.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:33.746 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:33.743+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:33.805 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:33.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:34.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:34.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:34.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:34.783+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:35.736 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:35.735+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:35.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:35.759+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:36.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:36.775+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:36.799 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:36.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:37.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:37.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:37.824 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:37.823+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:38.799 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:38.795+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:38.821 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:38.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:39.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:39.827+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:39.865 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:39.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:40.797 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:40.795+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:40.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:40.815+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:41.828 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:41.827+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:41.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:41.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:42.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:01:42.595 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:01:42.598 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:01:42.609 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6084,"stamp":"2024-01-12T01:01:41.856521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1978,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.627179+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:58:13.627179+0000","last_peered":"2024-01-12T00:58:13.627179+0000","last_clean":"2024-01-12T00:58:13.627179+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:58:13.627179+0000","last_undegraded":"2024-01-12T00:58:13.627179+0000","last_fullsized":"2024-01-12T00:58:13.627179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1711,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.924040+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:58:13.924040+0000","last_peered":"2024-01-12T00:58:13.924040+0000","last_clean":"2024-01-12T00:58:13.924040+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:58:13.924040+0000","last_undegraded":"2024-01-12T00:58:13.924040+0000","last_fullsized":"2024-01-12T00:58:13.924040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1923,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.218643+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:58:14.218643+0000","last_peered":"2024-01-12T00:58:14.218643+0000","last_clean":"2024-01-12T00:58:14.218643+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:58:14.218643+0000","last_undegraded":"2024-01-12T00:58:14.218643+0000","last_fullsized":"2024-01-12T00:58:14.218643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12869,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.512557+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:58:14.512557+0000","last_peered":"2024-01-12T00:58:14.512557+0000","last_clean":"2024-01-12T00:58:14.512557+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:58:14.512557+0000","last_undegraded":"2024-01-12T00:58:14.512557+0000","last_fullsized":"2024-01-12T00:58:14.512557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2142,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:58:14.811998+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:58:14.811998+0000","last_peered":"2024-01-12T00:58:14.811998+0000","last_clean":"2024-01-12T00:58:14.811998+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:58:14.811998+0000","last_undegraded":"2024-01-12T00:58:14.811998+0000","last_fullsized":"2024-01-12T00:58:14.811998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 391s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12353,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.100101+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:58:15.100101+0000","last_peered":"2024-01-12T00:58:15.100101+0000","last_clean":"2024-01-12T00:58:15.100101+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:58:15.100101+0000","last_undegraded":"2024-01-12T00:58:15.100101+0000","last_fullsized":"2024-01-12T00:58:15.100101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7504,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.389356+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:58:15.389356+0000","last_peered":"2024-01-12T00:58:15.389356+0000","last_clean":"2024-01-12T00:58:15.389356+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:58:15.389356+0000","last_undegraded":"2024-01-12T00:58:15.389356+0000","last_fullsized":"2024-01-12T00:58:15.389356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11781,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.676634+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:58:15.676634+0000","last_peered":"2024-01-12T00:58:15.676634+0000","last_clean":"2024-01-12T00:58:15.676634+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:58:15.676634+0000","last_undegraded":"2024-01-12T00:58:15.676634+0000","last_fullsized":"2024-01-12T00:58:15.676634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2519,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.969318+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:58:15.969318+0000","last_peered":"2024-01-12T00:58:15.969318+0000","last_clean":"2024-01-12T00:58:15.969318+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:58:15.969318+0000","last_undegraded":"2024-01-12T00:58:15.969318+0000","last_fullsized":"2024-01-12T00:58:15.969318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:16.279475+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:58:16.279475+0000","last_peered":"2024-01-12T00:58:16.279475+0000","last_clean":"2024-01-12T00:58:16.279475+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:58:16.279475+0000","last_undegraded":"2024-01-12T00:58:16.279475+0000","last_fullsized":"2024-01-12T00:58:16.279475+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740745,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.622,"5min":0.69499999999999995,"15min":0.93899999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.67800000000000005,"5min":0.70799999999999996,"15min":1.028},"last":0.59899999999999998}]},{"osd":1,"last update":"Fri Jan 12 01:00:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71599999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80500000000000005,"5min":0.84099999999999997,"15min":1.008},"last":0.69099999999999995}]}]},{"osd":1,"up_from":8,"seq":34359740743,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.61799999999999999,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.51200000000000001,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.77800000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jan 12 01:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46400000000000002,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.73399999999999999,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66600000000000004,"15min":0.66500000000000004},"min":{"1min":0.57299999999999995,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83599999999999997,"5min":1.0209999999999999,"15min":1.071},"last":0.64100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64400000000000002,"5min":0.748,"15min":0.79000000000000004},"last":0.502},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.51300000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.754,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.48799999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.91600000000000004},"last":0.63100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:01:42.610 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.610 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.610 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.611 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.611 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.611 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.611 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.611 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.611 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.612 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:01:42.612 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:01:42.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:42.839+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:42.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:42.911+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:43.871 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:43.871+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:43.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:43.883+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:44.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:44.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:44.853 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:44.851+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:45.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:45.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:45.881 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:45.879+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:46.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:46.855+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:46.921 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:46.919+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:47.822 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:47.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:47.906 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:47.903+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:48.812 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:48.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:48.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:48.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:49.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:49.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:49.919 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:49.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:50.856 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:50.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:50.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:50.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:51.836 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:51.834+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:51.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:51.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:52.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:52.854+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:52.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:52.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:53.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:53.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:53.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:53.954+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:54.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:54.842+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:55.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:54.998+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:55.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:55.802+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:55.987 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:55.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:56.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:56.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:56.992 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:56.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:57.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:57.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:58.022 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:58.018+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:58.861 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:58.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:59.010 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:59.006+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:59.814 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:01:59.810+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:01:59.998 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:01:59.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:00.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:00.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:00.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:00.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:01.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:01.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:02.003 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:02.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:02.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:02:02.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:02.710+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:02.904 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:02:02.908 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:02:02.918 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6094,"stamp":"2024-01-12T01:02:01.862308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003463"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1978,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.627179+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:58:13.627179+0000","last_peered":"2024-01-12T00:58:13.627179+0000","last_clean":"2024-01-12T00:58:13.627179+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:58:13.627179+0000","last_undegraded":"2024-01-12T00:58:13.627179+0000","last_fullsized":"2024-01-12T00:58:13.627179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1711,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.924040+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:58:13.924040+0000","last_peered":"2024-01-12T00:58:13.924040+0000","last_clean":"2024-01-12T00:58:13.924040+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:58:13.924040+0000","last_undegraded":"2024-01-12T00:58:13.924040+0000","last_fullsized":"2024-01-12T00:58:13.924040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1923,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.218643+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:58:14.218643+0000","last_peered":"2024-01-12T00:58:14.218643+0000","last_clean":"2024-01-12T00:58:14.218643+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:58:14.218643+0000","last_undegraded":"2024-01-12T00:58:14.218643+0000","last_fullsized":"2024-01-12T00:58:14.218643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12869,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.512557+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:58:14.512557+0000","last_peered":"2024-01-12T00:58:14.512557+0000","last_clean":"2024-01-12T00:58:14.512557+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:58:14.512557+0000","last_undegraded":"2024-01-12T00:58:14.512557+0000","last_fullsized":"2024-01-12T00:58:14.512557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2142,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:58:14.811998+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:58:14.811998+0000","last_peered":"2024-01-12T00:58:14.811998+0000","last_clean":"2024-01-12T00:58:14.811998+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:58:14.811998+0000","last_undegraded":"2024-01-12T00:58:14.811998+0000","last_fullsized":"2024-01-12T00:58:14.811998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 391s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12353,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.100101+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:58:15.100101+0000","last_peered":"2024-01-12T00:58:15.100101+0000","last_clean":"2024-01-12T00:58:15.100101+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:58:15.100101+0000","last_undegraded":"2024-01-12T00:58:15.100101+0000","last_fullsized":"2024-01-12T00:58:15.100101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7504,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.389356+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:58:15.389356+0000","last_peered":"2024-01-12T00:58:15.389356+0000","last_clean":"2024-01-12T00:58:15.389356+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:58:15.389356+0000","last_undegraded":"2024-01-12T00:58:15.389356+0000","last_fullsized":"2024-01-12T00:58:15.389356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11781,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.676634+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:58:15.676634+0000","last_peered":"2024-01-12T00:58:15.676634+0000","last_clean":"2024-01-12T00:58:15.676634+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:58:15.676634+0000","last_undegraded":"2024-01-12T00:58:15.676634+0000","last_fullsized":"2024-01-12T00:58:15.676634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2519,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.969318+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:58:15.969318+0000","last_peered":"2024-01-12T00:58:15.969318+0000","last_clean":"2024-01-12T00:58:15.969318+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:58:15.969318+0000","last_undegraded":"2024-01-12T00:58:15.969318+0000","last_fullsized":"2024-01-12T00:58:15.969318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:16.279475+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:58:16.279475+0000","last_peered":"2024-01-12T00:58:16.279475+0000","last_clean":"2024-01-12T00:58:16.279475+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:58:16.279475+0000","last_undegraded":"2024-01-12T00:58:16.279475+0000","last_fullsized":"2024-01-12T00:58:16.279475+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740749,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.432,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67800000000000005,"5min":0.68999999999999995,"15min":0.93899999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.49299999999999999,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":0.70799999999999996,"15min":1.028},"last":0.75}]},{"osd":1,"last update":"Fri Jan 12 01:01:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.45900000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.65500000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.68799999999999994},"min":{"1min":0.57099999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.008},"last":0.58099999999999996}]}]},{"osd":1,"up_from":8,"seq":34359740747,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.61799999999999999,"5min":0.68999999999999995,"15min":0.78500000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.51200000000000001,"5min":0.40400000000000003,"15min":0.39700000000000002},"max":{"1min":0.77800000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.875}]},{"osd":2,"last update":"Fri Jan 12 01:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":1.0209999999999999,"15min":1.071},"last":0.80800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64400000000000002,"5min":0.748,"15min":0.79000000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.51300000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.754,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.48799999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.91600000000000004},"last":0.63300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:02:02.919 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.919 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.919 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.920 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.920 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.920 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.920 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.920 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.921 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.921 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:02.921 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:02:03.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:02.998+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:03.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:03.754+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:03.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:03.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:04.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:04.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:04.926 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:04.922+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:05.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:05.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:05.969 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:05.966+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:06.712 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:06.710+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:06.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:06.934+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:07.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:07.738+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:07.918 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:07.914+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:08.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:08.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:08.942 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:08.938+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:09.787 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:09.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:09.902 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:09.898+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:10.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:10.782+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:10.912 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:10.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:11.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:11.754+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:11.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:11.958+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:12.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:12.790+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:12.971 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:12.970+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:13.797 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:13.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:13.964 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:13.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:14.841 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:14.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:14.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:14.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:15.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:15.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:15.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:15.990+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:16.823 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:16.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:16.965 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:16.962+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:17.854 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:17.850+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:17.945 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:17.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:18.827 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:18.826+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:18.922 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:18.918+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:19.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:19.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:19.948 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:19.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:20.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:20.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:20.935 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:20.930+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:21.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:21.886+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:21.943 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:21.942+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:22.878 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:22.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:22.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:02:22.979 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:22.978+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:23.213 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:02:23.215 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:02:23.226 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6104,"stamp":"2024-01-12T01:02:21.867648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1978,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.627179+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T00:58:13.627179+0000","last_peered":"2024-01-12T00:58:13.627179+0000","last_clean":"2024-01-12T00:58:13.627179+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T00:58:13.627179+0000","last_undegraded":"2024-01-12T00:58:13.627179+0000","last_fullsized":"2024-01-12T00:58:13.627179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1711,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:13.924040+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T00:58:13.924040+0000","last_peered":"2024-01-12T00:58:13.924040+0000","last_clean":"2024-01-12T00:58:13.924040+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T00:58:13.924040+0000","last_undegraded":"2024-01-12T00:58:13.924040+0000","last_fullsized":"2024-01-12T00:58:13.924040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1923,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.218643+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T00:58:14.218643+0000","last_peered":"2024-01-12T00:58:14.218643+0000","last_clean":"2024-01-12T00:58:14.218643+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T00:58:14.218643+0000","last_undegraded":"2024-01-12T00:58:14.218643+0000","last_fullsized":"2024-01-12T00:58:14.218643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12869,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:14.512557+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T00:58:14.512557+0000","last_peered":"2024-01-12T00:58:14.512557+0000","last_clean":"2024-01-12T00:58:14.512557+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T00:58:14.512557+0000","last_undegraded":"2024-01-12T00:58:14.512557+0000","last_fullsized":"2024-01-12T00:58:14.512557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2142,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T00:58:14.811998+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T00:58:14.811998+0000","last_peered":"2024-01-12T00:58:14.811998+0000","last_clean":"2024-01-12T00:58:14.811998+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T00:58:14.811998+0000","last_undegraded":"2024-01-12T00:58:14.811998+0000","last_fullsized":"2024-01-12T00:58:14.811998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 391s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12353,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.100101+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T00:58:15.100101+0000","last_peered":"2024-01-12T00:58:15.100101+0000","last_clean":"2024-01-12T00:58:15.100101+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T00:58:15.100101+0000","last_undegraded":"2024-01-12T00:58:15.100101+0000","last_fullsized":"2024-01-12T00:58:15.100101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7504,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.389356+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T00:58:15.389356+0000","last_peered":"2024-01-12T00:58:15.389356+0000","last_clean":"2024-01-12T00:58:15.389356+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T00:58:15.389356+0000","last_undegraded":"2024-01-12T00:58:15.389356+0000","last_fullsized":"2024-01-12T00:58:15.389356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11781,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.676634+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T00:58:15.676634+0000","last_peered":"2024-01-12T00:58:15.676634+0000","last_clean":"2024-01-12T00:58:15.676634+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T00:58:15.676634+0000","last_undegraded":"2024-01-12T00:58:15.676634+0000","last_fullsized":"2024-01-12T00:58:15.676634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2519,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:15.969318+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T00:58:15.969318+0000","last_peered":"2024-01-12T00:58:15.969318+0000","last_clean":"2024-01-12T00:58:15.969318+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T00:58:15.969318+0000","last_undegraded":"2024-01-12T00:58:15.969318+0000","last_fullsized":"2024-01-12T00:58:15.969318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:58:16.279475+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T00:58:16.279475+0000","last_peered":"2024-01-12T00:58:16.279475+0000","last_clean":"2024-01-12T00:58:16.279475+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T00:58:16.279475+0000","last_undegraded":"2024-01-12T00:58:16.279475+0000","last_fullsized":"2024-01-12T00:58:16.279475+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740753,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.432,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67800000000000005,"5min":0.68999999999999995,"15min":0.93899999999999995},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.49299999999999999,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":0.70799999999999996,"15min":1.028},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Jan 12 01:01:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.45900000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.65500000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.68799999999999994},"min":{"1min":0.57099999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.008},"last":0.59199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740751,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.66700000000000004,"5min":0.67600000000000005,"15min":0.78500000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63500000000000001,"15min":0.62},"min":{"1min":0.54700000000000004,"5min":0.47699999999999998,"15min":0.39700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":1.0209999999999999,"15min":1.071},"last":0.65200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359740754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64400000000000002,"5min":0.748,"15min":0.79000000000000004},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.51300000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.754,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.60199999999999998}]},{"osd":2,"last update":"Fri Jan 12 01:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.48799999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.91600000000000004},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:02:23.227 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.227 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.227 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.228 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.228 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.228 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.228 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.228 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.229 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.229 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:23.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2024-01-12T01:02:23.505 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2024-01-12T01:02:23.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2024-01-12T01:02:23.807 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2024-01-12T01:02:23.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2024-01-12T01:02:23.902 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:23.898+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:23.990 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:23.986+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:24.107 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2024-01-12T01:02:24.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2024-01-12T01:02:24.409 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2024-01-12T01:02:24.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2024-01-12T01:02:24.702 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2024-01-12T01:02:24.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2024-01-12T01:02:24.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:24.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:24.995 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:24.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:25.002 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-01-12T01:02:25.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2024-01-12T01:02:25.287 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2024-01-12T01:02:25.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2024-01-12T01:02:25.588 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2024-01-12T01:02:25.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2024-01-12T01:02:25.881 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:25.878+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:25.926 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2024-01-12T01:02:25.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2024-01-12T01:02:25.997 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:25.994+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:26.221 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-01-12T01:02:26.231 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:02:26.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:26.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:27.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:27.034+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:27.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:27.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:28.058 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:28.054+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:28.906 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:28.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:29.027 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:29.026+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:29.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:29.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:30.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:30.050+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:30.916 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:30.914+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:31.006 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:31.002+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:31.895 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:31.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:31.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:31.982+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:32.903 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:32.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:32.950 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:32.946+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:33.925 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:33.922+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:33.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:33.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:34.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:34.926+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:34.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:34.926+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:35.913 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:35.910+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:35.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:35.958+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:36.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:36.866+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:36.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:36.918+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:37.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:37.838+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:37.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:37.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:38.855 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:38.850+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:38.952 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:38.950+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:39.876 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:39.874+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:39.932 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:39.930+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:40.846 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:40.841+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:40.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:40.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:41.878 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:41.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:41.921 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:41.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:42.887 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:42.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:42.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:42.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:43.924 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:43.921+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:43.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:43.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:44.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:44.961+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:44.973 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:44.969+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:45.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:45.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:45.939 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:45.937+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:46.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:02:46.513 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:02:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:02:46.526 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6116,"stamp":"2024-01-12T01:02:45.873940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003224"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1979,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.654282+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:00:18.654282+0000","last_peered":"2024-01-12T01:00:18.654282+0000","last_clean":"2024-01-12T01:00:18.654282+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:00:18.654282+0000","last_undegraded":"2024-01-12T01:00:18.654282+0000","last_fullsized":"2024-01-12T01:00:18.654282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1712,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.954281+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:00:18.954281+0000","last_peered":"2024-01-12T01:00:18.954281+0000","last_clean":"2024-01-12T01:00:18.954281+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:00:18.954281+0000","last_undegraded":"2024-01-12T01:00:18.954281+0000","last_fullsized":"2024-01-12T01:00:18.954281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1924,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.256892+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:00:19.256892+0000","last_peered":"2024-01-12T01:00:19.256892+0000","last_clean":"2024-01-12T01:00:19.256892+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:00:19.256892+0000","last_undegraded":"2024-01-12T01:00:19.256892+0000","last_fullsized":"2024-01-12T01:00:19.256892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12870,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.550287+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:00:19.550287+0000","last_peered":"2024-01-12T01:00:19.550287+0000","last_clean":"2024-01-12T01:00:19.550287+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:00:19.550287+0000","last_undegraded":"2024-01-12T01:00:19.550287+0000","last_fullsized":"2024-01-12T01:00:19.550287+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2143,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:00:19.837750+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:00:19.837750+0000","last_peered":"2024-01-12T01:00:19.837750+0000","last_clean":"2024-01-12T01:00:19.837750+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:00:19.837750+0000","last_undegraded":"2024-01-12T01:00:19.837750+0000","last_fullsized":"2024-01-12T01:00:19.837750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12354,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.143283+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:00:20.143283+0000","last_peered":"2024-01-12T01:00:20.143283+0000","last_clean":"2024-01-12T01:00:20.143283+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:00:20.143283+0000","last_undegraded":"2024-01-12T01:00:20.143283+0000","last_fullsized":"2024-01-12T01:00:20.143283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7505,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.447228+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:00:20.447228+0000","last_peered":"2024-01-12T01:00:20.447228+0000","last_clean":"2024-01-12T01:00:20.447228+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:00:20.447228+0000","last_undegraded":"2024-01-12T01:00:20.447228+0000","last_fullsized":"2024-01-12T01:00:20.447228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11782,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.749433+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:00:20.749433+0000","last_peered":"2024-01-12T01:00:20.749433+0000","last_clean":"2024-01-12T01:00:20.749433+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:00:20.749433+0000","last_undegraded":"2024-01-12T01:00:20.749433+0000","last_fullsized":"2024-01-12T01:00:20.749433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2520,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.051510+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:00:21.051510+0000","last_peered":"2024-01-12T01:00:21.051510+0000","last_clean":"2024-01-12T01:00:21.051510+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:00:21.051510+0000","last_undegraded":"2024-01-12T01:00:21.051510+0000","last_fullsized":"2024-01-12T01:00:21.051510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.348692+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:00:21.348692+0000","last_peered":"2024-01-12T01:00:21.348692+0000","last_clean":"2024-01-12T01:00:21.348692+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:00:21.348692+0000","last_undegraded":"2024-01-12T01:00:21.348692+0000","last_fullsized":"2024-01-12T01:00:21.348692+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740758,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.432,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67800000000000005,"5min":0.68999999999999995,"15min":0.93899999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.49299999999999999,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":0.70799999999999996,"15min":1.028},"last":0.63800000000000001}]},{"osd":1,"last update":"Fri Jan 12 01:01:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.45900000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.65500000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68300000000000005,"15min":0.68799999999999994},"min":{"1min":0.57099999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.008},"last":0.77000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740756,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.66700000000000004,"5min":0.67600000000000005,"15min":0.78500000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63500000000000001,"15min":0.62},"min":{"1min":0.54700000000000004,"5min":0.47699999999999998,"15min":0.39700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Jan 12 01:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.439,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":1.0209999999999999,"15min":1.071},"last":0.76100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.46200000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.79000000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.502,"5min":0.42799999999999999,"15min":0.40600000000000003},"max":{"1min":0.82299999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.71599999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.59299999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":0.91600000000000004},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:02:46.528 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.528 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.528 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.528 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.528 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.529 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.529 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.529 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.529 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.530 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:02:46.530 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:02:46.904 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:46.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:46.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:46.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:47.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:47.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:47.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:47.961+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:48.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:48.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:49.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:48.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:49.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:49.953+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:50.042 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:50.037+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:50.978 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:50.973+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:51.045 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:51.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:51.959 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:51.957+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:52.036 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:52.033+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:52.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:52.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:52.995 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:52.993+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:53.903 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:53.901+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:54.044 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:54.041+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:54.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:54.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:55.021 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:55.017+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:55.894 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:55.889+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:56.043 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:56.037+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:56.861 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:56.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:57.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:56.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:57.860 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:57.857+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:57.977 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:57.973+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:58.869 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:58.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:59.000 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:58.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:59.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:02:59.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:02:59.975 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:02:59.973+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:00.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:00.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:00.931 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:00.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:01.760 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:01.757+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:01.917 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:01.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:02.747 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:02.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:02.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:02.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:03.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:03.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:03.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:03.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:04.767 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:04.765+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:04.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:04.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:05.806 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:05.801+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:05.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:05.889+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:06.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:03:06.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:06.813+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:06.820 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:03:06.822 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:03:06.832 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6126,"stamp":"2024-01-12T01:03:05.879304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003138"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1979,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.654282+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:00:18.654282+0000","last_peered":"2024-01-12T01:00:18.654282+0000","last_clean":"2024-01-12T01:00:18.654282+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:00:18.654282+0000","last_undegraded":"2024-01-12T01:00:18.654282+0000","last_fullsized":"2024-01-12T01:00:18.654282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1712,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.954281+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:00:18.954281+0000","last_peered":"2024-01-12T01:00:18.954281+0000","last_clean":"2024-01-12T01:00:18.954281+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:00:18.954281+0000","last_undegraded":"2024-01-12T01:00:18.954281+0000","last_fullsized":"2024-01-12T01:00:18.954281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1924,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.256892+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:00:19.256892+0000","last_peered":"2024-01-12T01:00:19.256892+0000","last_clean":"2024-01-12T01:00:19.256892+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:00:19.256892+0000","last_undegraded":"2024-01-12T01:00:19.256892+0000","last_fullsized":"2024-01-12T01:00:19.256892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12870,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.550287+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:00:19.550287+0000","last_peered":"2024-01-12T01:00:19.550287+0000","last_clean":"2024-01-12T01:00:19.550287+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:00:19.550287+0000","last_undegraded":"2024-01-12T01:00:19.550287+0000","last_fullsized":"2024-01-12T01:00:19.550287+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2143,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:00:19.837750+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:00:19.837750+0000","last_peered":"2024-01-12T01:00:19.837750+0000","last_clean":"2024-01-12T01:00:19.837750+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:00:19.837750+0000","last_undegraded":"2024-01-12T01:00:19.837750+0000","last_fullsized":"2024-01-12T01:00:19.837750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12354,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.143283+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:00:20.143283+0000","last_peered":"2024-01-12T01:00:20.143283+0000","last_clean":"2024-01-12T01:00:20.143283+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:00:20.143283+0000","last_undegraded":"2024-01-12T01:00:20.143283+0000","last_fullsized":"2024-01-12T01:00:20.143283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7505,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.447228+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:00:20.447228+0000","last_peered":"2024-01-12T01:00:20.447228+0000","last_clean":"2024-01-12T01:00:20.447228+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:00:20.447228+0000","last_undegraded":"2024-01-12T01:00:20.447228+0000","last_fullsized":"2024-01-12T01:00:20.447228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11782,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.749433+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:00:20.749433+0000","last_peered":"2024-01-12T01:00:20.749433+0000","last_clean":"2024-01-12T01:00:20.749433+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:00:20.749433+0000","last_undegraded":"2024-01-12T01:00:20.749433+0000","last_fullsized":"2024-01-12T01:00:20.749433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2520,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.051510+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:00:21.051510+0000","last_peered":"2024-01-12T01:00:21.051510+0000","last_clean":"2024-01-12T01:00:21.051510+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:00:21.051510+0000","last_undegraded":"2024-01-12T01:00:21.051510+0000","last_fullsized":"2024-01-12T01:00:21.051510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.348692+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:00:21.348692+0000","last_peered":"2024-01-12T01:00:21.348692+0000","last_clean":"2024-01-12T01:00:21.348692+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:00:21.348692+0000","last_undegraded":"2024-01-12T01:00:21.348692+0000","last_fullsized":"2024-01-12T01:00:21.348692+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740762,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:02:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.46500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.63600000000000001,"5min":0.68999999999999995,"15min":0.93899999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.57799999999999996,"15min":0.56899999999999995},"min":{"1min":0.48499999999999999,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":1.028},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Jan 12 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.48299999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.67700000000000005,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77000000000000002,"5min":0.92700000000000005,"15min":1.008},"last":0.66800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740760,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.66700000000000004,"5min":0.67600000000000005,"15min":0.78500000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63500000000000001,"15min":0.62},"min":{"1min":0.54700000000000004,"5min":0.47699999999999998,"15min":0.39700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61499999999999999}]},{"osd":2,"last update":"Fri Jan 12 01:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60499999999999998,"15min":0.58999999999999997},"min":{"1min":0.55800000000000005,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68300000000000005,"15min":0.67200000000000004},"min":{"1min":0.65200000000000002,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.73499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.46200000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.79000000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.502,"5min":0.42799999999999999,"15min":0.40600000000000003},"max":{"1min":0.82299999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.71599999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.59299999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":0.91600000000000004},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:03:06.834 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.834 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.834 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.834 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.834 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.834 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.835 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.835 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.835 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.835 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:06.835 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:03:06.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:06.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:07.863 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:07.861+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:07.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:07.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:08.848 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:08.845+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:08.877 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:08.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:09.822 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:09.817+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:09.884 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:09.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:10.837 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:10.833+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:10.886 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:10.881+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:11.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:11.789+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:11.937 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:11.933+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:12.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:12.781+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:12.983 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:12.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:13.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:13.777+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:13.994 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:13.989+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:14.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:14.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:14.965 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:14.961+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:15.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:15.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:15.986 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:15.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:16.827 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:16.825+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:16.985 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:16.981+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:17.877 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:17.873+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:17.988 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:17.985+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:18.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:18.865+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:18.953 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:18.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:19.882 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:19.877+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:19.940 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:19.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:20.896 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:20.893+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:20.955 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:20.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:21.923 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:21.917+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:22.001 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:21.997+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:22.944 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:22.941+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:22.957 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:22.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:23.929 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:23.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:23.974 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:23.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:24.909 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:24.905+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:24.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:24.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:25.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:25.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:25.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:25.925+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:26.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:03:26.938 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:26.933+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:26.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:26.961+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:27.129 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:03:27.132 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:03:27.142 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6136,"stamp":"2024-01-12T01:03:25.884985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003222"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1979,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.654282+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:00:18.654282+0000","last_peered":"2024-01-12T01:00:18.654282+0000","last_clean":"2024-01-12T01:00:18.654282+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:00:18.654282+0000","last_undegraded":"2024-01-12T01:00:18.654282+0000","last_fullsized":"2024-01-12T01:00:18.654282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1712,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.954281+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:00:18.954281+0000","last_peered":"2024-01-12T01:00:18.954281+0000","last_clean":"2024-01-12T01:00:18.954281+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:00:18.954281+0000","last_undegraded":"2024-01-12T01:00:18.954281+0000","last_fullsized":"2024-01-12T01:00:18.954281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1924,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.256892+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:00:19.256892+0000","last_peered":"2024-01-12T01:00:19.256892+0000","last_clean":"2024-01-12T01:00:19.256892+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:00:19.256892+0000","last_undegraded":"2024-01-12T01:00:19.256892+0000","last_fullsized":"2024-01-12T01:00:19.256892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12870,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.550287+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:00:19.550287+0000","last_peered":"2024-01-12T01:00:19.550287+0000","last_clean":"2024-01-12T01:00:19.550287+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:00:19.550287+0000","last_undegraded":"2024-01-12T01:00:19.550287+0000","last_fullsized":"2024-01-12T01:00:19.550287+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2143,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:00:19.837750+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:00:19.837750+0000","last_peered":"2024-01-12T01:00:19.837750+0000","last_clean":"2024-01-12T01:00:19.837750+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:00:19.837750+0000","last_undegraded":"2024-01-12T01:00:19.837750+0000","last_fullsized":"2024-01-12T01:00:19.837750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12354,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.143283+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:00:20.143283+0000","last_peered":"2024-01-12T01:00:20.143283+0000","last_clean":"2024-01-12T01:00:20.143283+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:00:20.143283+0000","last_undegraded":"2024-01-12T01:00:20.143283+0000","last_fullsized":"2024-01-12T01:00:20.143283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7505,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.447228+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:00:20.447228+0000","last_peered":"2024-01-12T01:00:20.447228+0000","last_clean":"2024-01-12T01:00:20.447228+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:00:20.447228+0000","last_undegraded":"2024-01-12T01:00:20.447228+0000","last_fullsized":"2024-01-12T01:00:20.447228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11782,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.749433+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:00:20.749433+0000","last_peered":"2024-01-12T01:00:20.749433+0000","last_clean":"2024-01-12T01:00:20.749433+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:00:20.749433+0000","last_undegraded":"2024-01-12T01:00:20.749433+0000","last_fullsized":"2024-01-12T01:00:20.749433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2520,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.051510+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:00:21.051510+0000","last_peered":"2024-01-12T01:00:21.051510+0000","last_clean":"2024-01-12T01:00:21.051510+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:00:21.051510+0000","last_undegraded":"2024-01-12T01:00:21.051510+0000","last_fullsized":"2024-01-12T01:00:21.051510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.348692+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:00:21.348692+0000","last_peered":"2024-01-12T01:00:21.348692+0000","last_clean":"2024-01-12T01:00:21.348692+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:00:21.348692+0000","last_undegraded":"2024-01-12T01:00:21.348692+0000","last_fullsized":"2024-01-12T01:00:21.348692+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740766,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:02:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.46500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.63600000000000001,"5min":0.68999999999999995,"15min":0.93899999999999995},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.57799999999999996,"15min":0.56899999999999995},"min":{"1min":0.48499999999999999,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":1.028},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Jan 12 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.48299999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.67700000000000005,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77000000000000002,"5min":0.92700000000000005,"15min":1.008},"last":0.69699999999999995}]}]},{"osd":1,"up_from":8,"seq":34359740764,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:03:16 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.35099999999999998},"max":{"1min":0.56499999999999995,"5min":0.66700000000000004,"15min":0.78500000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.39700000000000002},"max":{"1min":0.76200000000000001,"5min":0.875,"15min":0.875},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Jan 12 01:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60499999999999998,"15min":0.58999999999999997},"min":{"1min":0.55800000000000005,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68300000000000005,"15min":0.67200000000000004},"min":{"1min":0.65200000000000002,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.76800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359740767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.46200000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.79000000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.502,"5min":0.42799999999999999,"15min":0.40600000000000003},"max":{"1min":0.82299999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Jan 12 01:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.71599999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.59299999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":0.91600000000000004},"last":0.69799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:03:27.143 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.143 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.144 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.144 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.144 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.144 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.144 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.145 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.145 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.145 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:27.145 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:03:27.912 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:27.909+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:27.953 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:27.949+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:28.889 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:28.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:28.955 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:28.953+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:29.879 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:29.873+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:30.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:29.997+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:30.839 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:30.837+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:31.049 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:31.045+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:31.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:31.821+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:32.004 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:32.001+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:32.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:32.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:32.994 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:32.988+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:33.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:33.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:33.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:33.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:34.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:34.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:35.000 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:34.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:35.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:35.812+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:35.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:35.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:36.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:36.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:37.033 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:37.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:37.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:37.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:38.043 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:38.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:38.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:38.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:39.055 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:39.052+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:39.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:39.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:40.050 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:40.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:40.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:40.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:41.039 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:41.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:41.830 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:41.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:42.051 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:42.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:42.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:42.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:43.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:43.076+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:43.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:43.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:44.047 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:44.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:44.857 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:44.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:45.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:45.060+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:45.900 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:45.896+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:46.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:46.088+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:46.883 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:46.880+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:47.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:47.076+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:47.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:03:47.439 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:03:47.442 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:03:47.452 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6146,"stamp":"2024-01-12T01:03:45.890289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003050"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1979,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.654282+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:00:18.654282+0000","last_peered":"2024-01-12T01:00:18.654282+0000","last_clean":"2024-01-12T01:00:18.654282+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:00:18.654282+0000","last_undegraded":"2024-01-12T01:00:18.654282+0000","last_fullsized":"2024-01-12T01:00:18.654282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1712,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.954281+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:00:18.954281+0000","last_peered":"2024-01-12T01:00:18.954281+0000","last_clean":"2024-01-12T01:00:18.954281+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:00:18.954281+0000","last_undegraded":"2024-01-12T01:00:18.954281+0000","last_fullsized":"2024-01-12T01:00:18.954281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1924,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.256892+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:00:19.256892+0000","last_peered":"2024-01-12T01:00:19.256892+0000","last_clean":"2024-01-12T01:00:19.256892+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:00:19.256892+0000","last_undegraded":"2024-01-12T01:00:19.256892+0000","last_fullsized":"2024-01-12T01:00:19.256892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12870,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.550287+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:00:19.550287+0000","last_peered":"2024-01-12T01:00:19.550287+0000","last_clean":"2024-01-12T01:00:19.550287+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:00:19.550287+0000","last_undegraded":"2024-01-12T01:00:19.550287+0000","last_fullsized":"2024-01-12T01:00:19.550287+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2143,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:00:19.837750+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:00:19.837750+0000","last_peered":"2024-01-12T01:00:19.837750+0000","last_clean":"2024-01-12T01:00:19.837750+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:00:19.837750+0000","last_undegraded":"2024-01-12T01:00:19.837750+0000","last_fullsized":"2024-01-12T01:00:19.837750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12354,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.143283+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:00:20.143283+0000","last_peered":"2024-01-12T01:00:20.143283+0000","last_clean":"2024-01-12T01:00:20.143283+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:00:20.143283+0000","last_undegraded":"2024-01-12T01:00:20.143283+0000","last_fullsized":"2024-01-12T01:00:20.143283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7505,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.447228+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:00:20.447228+0000","last_peered":"2024-01-12T01:00:20.447228+0000","last_clean":"2024-01-12T01:00:20.447228+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:00:20.447228+0000","last_undegraded":"2024-01-12T01:00:20.447228+0000","last_fullsized":"2024-01-12T01:00:20.447228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11782,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.749433+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:00:20.749433+0000","last_peered":"2024-01-12T01:00:20.749433+0000","last_clean":"2024-01-12T01:00:20.749433+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:00:20.749433+0000","last_undegraded":"2024-01-12T01:00:20.749433+0000","last_fullsized":"2024-01-12T01:00:20.749433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2520,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.051510+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:00:21.051510+0000","last_peered":"2024-01-12T01:00:21.051510+0000","last_clean":"2024-01-12T01:00:21.051510+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:00:21.051510+0000","last_undegraded":"2024-01-12T01:00:21.051510+0000","last_fullsized":"2024-01-12T01:00:21.051510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.348692+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:00:21.348692+0000","last_peered":"2024-01-12T01:00:21.348692+0000","last_clean":"2024-01-12T01:00:21.348692+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:00:21.348692+0000","last_undegraded":"2024-01-12T01:00:21.348692+0000","last_fullsized":"2024-01-12T01:00:21.348692+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740770,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:02:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.46500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.63600000000000001,"5min":0.68999999999999995,"15min":0.93899999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.57799999999999996,"15min":0.56899999999999995},"min":{"1min":0.48499999999999999,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":1.028},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri Jan 12 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.48299999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.67700000000000005,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77000000000000002,"5min":0.92700000000000005,"15min":1.008},"last":0.73699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359740768,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:03:16 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.35099999999999998},"max":{"1min":0.56499999999999995,"5min":0.66700000000000004,"15min":0.78500000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.39700000000000002},"max":{"1min":0.76200000000000001,"5min":0.875,"15min":0.875},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60499999999999998,"15min":0.58999999999999997},"min":{"1min":0.55800000000000005,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.627},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68300000000000005,"15min":0.67200000000000004},"min":{"1min":0.65200000000000002,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.748}]}]},{"osd":0,"up_from":8,"seq":34359740771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.40500000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.68899999999999995,"5min":0.751,"15min":0.79000000000000004},"last":0.504},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.49199999999999999,"5min":0.45300000000000001,"15min":0.439},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64500000000000002,"15min":0.64200000000000002},"min":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.754,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:03:47.453 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.453 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.454 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.454 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.454 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.454 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.454 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.455 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.455 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.455 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:03:47.455 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:03:47.929 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:47.924+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:48.034 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:48.028+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:48.969 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:48.964+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:48.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:48.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:49.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:49.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:49.974 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:49.968+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:50.928 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:50.924+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:50.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:50.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:51.956 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:51.952+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:52.003 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:52.000+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:52.981 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:52.976+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:52.996 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:52.992+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:53.963 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:53.960+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:53.966 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:53.960+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:54.962 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:54.956+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:54.991 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:54.984+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:55.935 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:55.932+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:56.009 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:56.004+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:56.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:56.884+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:56.961 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:56.956+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:57.858 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:57.852+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:57.980 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:57.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:58.832 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:58.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:59.001 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:58.996+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:59.786 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:03:59.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:03:59.986 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:03:59.980+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:00.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:00.820+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:01.029 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:01.024+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:01.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:01.860+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:01.988 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:01.984+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:02.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:02.840+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:02.983 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:02.976+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:03.843 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:03.836+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:03.989 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:03.984+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:04.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:04.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:05.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:05.032+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:05.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:05.768+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:06.066 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:06.060+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:06.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:06.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:07.043 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:07.036+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:07.456 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:04:07.753 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:04:07.756 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:04:07.759 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:07.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:07.766 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6156,"stamp":"2024-01-12T01:04:05.895792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003238"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1979,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.654282+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:00:18.654282+0000","last_peered":"2024-01-12T01:00:18.654282+0000","last_clean":"2024-01-12T01:00:18.654282+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:00:18.654282+0000","last_undegraded":"2024-01-12T01:00:18.654282+0000","last_fullsized":"2024-01-12T01:00:18.654282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1712,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.954281+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:00:18.954281+0000","last_peered":"2024-01-12T01:00:18.954281+0000","last_clean":"2024-01-12T01:00:18.954281+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:00:18.954281+0000","last_undegraded":"2024-01-12T01:00:18.954281+0000","last_fullsized":"2024-01-12T01:00:18.954281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1924,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.256892+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:00:19.256892+0000","last_peered":"2024-01-12T01:00:19.256892+0000","last_clean":"2024-01-12T01:00:19.256892+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:00:19.256892+0000","last_undegraded":"2024-01-12T01:00:19.256892+0000","last_fullsized":"2024-01-12T01:00:19.256892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12870,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.550287+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:00:19.550287+0000","last_peered":"2024-01-12T01:00:19.550287+0000","last_clean":"2024-01-12T01:00:19.550287+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:00:19.550287+0000","last_undegraded":"2024-01-12T01:00:19.550287+0000","last_fullsized":"2024-01-12T01:00:19.550287+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2143,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:00:19.837750+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:00:19.837750+0000","last_peered":"2024-01-12T01:00:19.837750+0000","last_clean":"2024-01-12T01:00:19.837750+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:00:19.837750+0000","last_undegraded":"2024-01-12T01:00:19.837750+0000","last_fullsized":"2024-01-12T01:00:19.837750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12354,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.143283+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:00:20.143283+0000","last_peered":"2024-01-12T01:00:20.143283+0000","last_clean":"2024-01-12T01:00:20.143283+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:00:20.143283+0000","last_undegraded":"2024-01-12T01:00:20.143283+0000","last_fullsized":"2024-01-12T01:00:20.143283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7505,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.447228+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:00:20.447228+0000","last_peered":"2024-01-12T01:00:20.447228+0000","last_clean":"2024-01-12T01:00:20.447228+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:00:20.447228+0000","last_undegraded":"2024-01-12T01:00:20.447228+0000","last_fullsized":"2024-01-12T01:00:20.447228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11782,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.749433+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:00:20.749433+0000","last_peered":"2024-01-12T01:00:20.749433+0000","last_clean":"2024-01-12T01:00:20.749433+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:00:20.749433+0000","last_undegraded":"2024-01-12T01:00:20.749433+0000","last_fullsized":"2024-01-12T01:00:20.749433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2520,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.051510+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:00:21.051510+0000","last_peered":"2024-01-12T01:00:21.051510+0000","last_clean":"2024-01-12T01:00:21.051510+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:00:21.051510+0000","last_undegraded":"2024-01-12T01:00:21.051510+0000","last_fullsized":"2024-01-12T01:00:21.051510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.348692+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:00:21.348692+0000","last_peered":"2024-01-12T01:00:21.348692+0000","last_clean":"2024-01-12T01:00:21.348692+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:00:21.348692+0000","last_undegraded":"2024-01-12T01:00:21.348692+0000","last_fullsized":"2024-01-12T01:00:21.348692+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740774,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.47399999999999998,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.71199999999999997,"5min":0.754,"15min":1.028},"last":0.61699999999999999}]},{"osd":1,"last update":"Fri Jan 12 01:03:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":0.79600000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67600000000000005,"15min":0.68500000000000005},"min":{"1min":0.59099999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79200000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.69199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359740772,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:03:16 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.35099999999999998},"max":{"1min":0.56499999999999995,"5min":0.66700000000000004,"15min":0.78500000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.39700000000000002},"max":{"1min":0.76200000000000001,"5min":0.875,"15min":0.875},"last":0.64000000000000001}]},{"osd":2,"last update":"Fri Jan 12 01:03:56 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61699999999999999,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.746,"5min":0.877,"15min":0.877},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.67600000000000005},"min":{"1min":0.65400000000000003,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.80100000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.80500000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.40500000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.68899999999999995,"5min":0.751,"15min":0.79000000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.61299999999999999}]},{"osd":2,"last update":"Fri Jan 12 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.49199999999999999,"5min":0.45300000000000001,"15min":0.439},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64500000000000002,"15min":0.64200000000000002},"min":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.754,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:04:07.767 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.768 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.768 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.768 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.768 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.768 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.769 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.769 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.769 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.769 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:07.769 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:04:08.046 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:08.040+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:08.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:08.768+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:09.053 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:09.048+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:09.770 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:09.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:10.093 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:10.092+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:10.771 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:10.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:11.078 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:11.076+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:11.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:11.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:12.037 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:12.036+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:12.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:12.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:13.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:13.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:13.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:13.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:14.056 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:14.056+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:14.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:14.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:15.045 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:15.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:15.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:15.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:16.044 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:16.044+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:16.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:16.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:17.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:17.092+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:17.741 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:17.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:18.067 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:18.064+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:18.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:18.724+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:19.098 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:19.096+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:19.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:19.732+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:20.124 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:20.124+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:20.754 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:20.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:21.128 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:21.128+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:21.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:21.768+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:22.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:22.136+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:22.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:22.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:23.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:23.132+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:23.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:23.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:24.109 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:24.108+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:24.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:24.732+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:25.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:25.152+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:25.697 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:25.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:26.123 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:26.119+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:26.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:26.683+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:27.171 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:27.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:27.694 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:27.691+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:27.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:04:28.060 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:04:28.063 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:04:28.074 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6167,"stamp":"2024-01-12T01:04:27.901725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003242"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1979,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.654282+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:00:18.654282+0000","last_peered":"2024-01-12T01:00:18.654282+0000","last_clean":"2024-01-12T01:00:18.654282+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:00:18.654282+0000","last_undegraded":"2024-01-12T01:00:18.654282+0000","last_fullsized":"2024-01-12T01:00:18.654282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1712,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:18.954281+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:00:18.954281+0000","last_peered":"2024-01-12T01:00:18.954281+0000","last_clean":"2024-01-12T01:00:18.954281+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:00:18.954281+0000","last_undegraded":"2024-01-12T01:00:18.954281+0000","last_fullsized":"2024-01-12T01:00:18.954281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1924,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.256892+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:00:19.256892+0000","last_peered":"2024-01-12T01:00:19.256892+0000","last_clean":"2024-01-12T01:00:19.256892+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:00:19.256892+0000","last_undegraded":"2024-01-12T01:00:19.256892+0000","last_fullsized":"2024-01-12T01:00:19.256892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12870,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:19.550287+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:00:19.550287+0000","last_peered":"2024-01-12T01:00:19.550287+0000","last_clean":"2024-01-12T01:00:19.550287+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:00:19.550287+0000","last_undegraded":"2024-01-12T01:00:19.550287+0000","last_fullsized":"2024-01-12T01:00:19.550287+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2143,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:00:19.837750+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:00:19.837750+0000","last_peered":"2024-01-12T01:00:19.837750+0000","last_clean":"2024-01-12T01:00:19.837750+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:00:19.837750+0000","last_undegraded":"2024-01-12T01:00:19.837750+0000","last_fullsized":"2024-01-12T01:00:19.837750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12354,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.143283+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:00:20.143283+0000","last_peered":"2024-01-12T01:00:20.143283+0000","last_clean":"2024-01-12T01:00:20.143283+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:00:20.143283+0000","last_undegraded":"2024-01-12T01:00:20.143283+0000","last_fullsized":"2024-01-12T01:00:20.143283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7505,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.447228+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:00:20.447228+0000","last_peered":"2024-01-12T01:00:20.447228+0000","last_clean":"2024-01-12T01:00:20.447228+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:00:20.447228+0000","last_undegraded":"2024-01-12T01:00:20.447228+0000","last_fullsized":"2024-01-12T01:00:20.447228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11782,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:20.749433+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:00:20.749433+0000","last_peered":"2024-01-12T01:00:20.749433+0000","last_clean":"2024-01-12T01:00:20.749433+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:00:20.749433+0000","last_undegraded":"2024-01-12T01:00:20.749433+0000","last_fullsized":"2024-01-12T01:00:20.749433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2520,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.051510+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:00:21.051510+0000","last_peered":"2024-01-12T01:00:21.051510+0000","last_clean":"2024-01-12T01:00:21.051510+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:00:21.051510+0000","last_undegraded":"2024-01-12T01:00:21.051510+0000","last_fullsized":"2024-01-12T01:00:21.051510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:00:21.348692+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:00:21.348692+0000","last_peered":"2024-01-12T01:00:21.348692+0000","last_clean":"2024-01-12T01:00:21.348692+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:00:21.348692+0000","last_undegraded":"2024-01-12T01:00:21.348692+0000","last_fullsized":"2024-01-12T01:00:21.348692+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740778,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.47399999999999998,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.71199999999999997,"5min":0.754,"15min":1.028},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Jan 12 01:03:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":0.79600000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67600000000000005,"15min":0.68500000000000005},"min":{"1min":0.59099999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79200000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.755}]}]},{"osd":1,"up_from":8,"seq":34359740776,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.44800000000000001,"5min":0.40799999999999997,"15min":0.35099999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.78300000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.54100000000000004,"5min":0.51200000000000001,"15min":0.39700000000000002},"max":{"1min":0.81399999999999995,"5min":0.875,"15min":0.875},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:03:56 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61699999999999999,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.746,"5min":0.877,"15min":0.877},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.67600000000000005},"min":{"1min":0.65400000000000003,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.80100000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.70099999999999996}]}]},{"osd":0,"up_from":8,"seq":34359740779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.40500000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.68899999999999995,"5min":0.751,"15min":0.79000000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.49199999999999999,"5min":0.45300000000000001,"15min":0.439},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64500000000000002,"15min":0.64200000000000002},"min":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.754,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:04:28.075 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.075 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.076 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.076 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.076 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.076 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.076 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.076 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.077 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.077 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:28.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2024-01-12T01:04:28.133 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:28.131+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:28.360 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2024-01-12T01:04:28.369 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2024-01-12T01:04:28.653 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2024-01-12T01:04:28.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2024-01-12T01:04:28.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:28.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:28.944 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2024-01-12T01:04:28.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2024-01-12T01:04:29.182 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:29.179+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:29.239 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2024-01-12T01:04:29.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2024-01-12T01:04:29.531 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2024-01-12T01:04:29.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2024-01-12T01:04:29.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:29.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:29.823 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-01-12T01:04:29.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2024-01-12T01:04:30.130 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2024-01-12T01:04:30.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2024-01-12T01:04:30.199 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:30.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:30.426 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2024-01-12T01:04:30.436 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2024-01-12T01:04:30.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:30.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:30.725 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2024-01-12T01:04:30.735 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2024-01-12T01:04:31.015 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-01-12T01:04:31.025 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:04:31.177 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:31.175+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:31.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:31.699+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:32.185 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:32.183+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:32.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:32.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:33.208 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:33.207+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:33.712 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:33.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:34.201 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:34.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:34.687 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:34.687+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:35.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:35.179+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:35.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:35.679+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:36.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:36.179+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:36.679 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:36.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:37.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:37.183+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:37.677 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:37.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:38.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:38.219+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:38.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:38.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:39.258 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:39.255+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:39.692 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:39.691+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:40.232 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:40.231+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:40.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:40.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:41.202 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:41.199+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:41.709 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:41.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:42.226 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:42.223+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:42.730 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:42.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:43.217 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:43.215+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:43.715 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:43.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:44.192 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:44.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:44.706 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:44.703+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:45.204 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:45.203+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:45.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:45.731+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:46.159 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:46.159+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:46.691 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:46.691+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:47.147 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:47.147+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:47.676 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:47.675+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:48.114 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:48.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:48.685 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:48.683+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:49.082 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:49.079+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:49.646 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:49.643+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:50.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:50.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:50.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:50.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:51.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:04:51.077 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:51.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:04:51.317 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:04:51.327 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6178,"stamp":"2024-01-12T01:04:49.908087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003611"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1980,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.503197+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:02:23.503197+0000","last_peered":"2024-01-12T01:02:23.503197+0000","last_clean":"2024-01-12T01:02:23.503197+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:02:23.503197+0000","last_undegraded":"2024-01-12T01:02:23.503197+0000","last_fullsized":"2024-01-12T01:02:23.503197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1713,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.805441+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:02:23.805441+0000","last_peered":"2024-01-12T01:02:23.805441+0000","last_clean":"2024-01-12T01:02:23.805441+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:02:23.805441+0000","last_undegraded":"2024-01-12T01:02:23.805441+0000","last_fullsized":"2024-01-12T01:02:23.805441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1925,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.105419+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:02:24.105419+0000","last_peered":"2024-01-12T01:02:24.105419+0000","last_clean":"2024-01-12T01:02:24.105419+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:02:24.105419+0000","last_undegraded":"2024-01-12T01:02:24.105419+0000","last_fullsized":"2024-01-12T01:02:24.105419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12871,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.407169+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:02:24.407169+0000","last_peered":"2024-01-12T01:02:24.407169+0000","last_clean":"2024-01-12T01:02:24.407169+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:02:24.407169+0000","last_undegraded":"2024-01-12T01:02:24.407169+0000","last_fullsized":"2024-01-12T01:02:24.407169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2144,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:02:24.701024+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:02:24.701024+0000","last_peered":"2024-01-12T01:02:24.701024+0000","last_clean":"2024-01-12T01:02:24.701024+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:02:24.701024+0000","last_undegraded":"2024-01-12T01:02:24.701024+0000","last_fullsized":"2024-01-12T01:02:24.701024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12355,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.000209+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:02:25.000209+0000","last_peered":"2024-01-12T01:02:25.000209+0000","last_clean":"2024-01-12T01:02:25.000209+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:02:25.000209+0000","last_undegraded":"2024-01-12T01:02:25.000209+0000","last_fullsized":"2024-01-12T01:02:25.000209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7506,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.285268+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:02:25.285268+0000","last_peered":"2024-01-12T01:02:25.285268+0000","last_clean":"2024-01-12T01:02:25.285268+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:02:25.285268+0000","last_undegraded":"2024-01-12T01:02:25.285268+0000","last_fullsized":"2024-01-12T01:02:25.285268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11783,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.586098+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:02:25.586098+0000","last_peered":"2024-01-12T01:02:25.586098+0000","last_clean":"2024-01-12T01:02:25.586098+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:02:25.586098+0000","last_undegraded":"2024-01-12T01:02:25.586098+0000","last_fullsized":"2024-01-12T01:02:25.586098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2521,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.921978+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:02:25.921978+0000","last_peered":"2024-01-12T01:02:25.921978+0000","last_clean":"2024-01-12T01:02:25.921978+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:02:25.921978+0000","last_undegraded":"2024-01-12T01:02:25.921978+0000","last_fullsized":"2024-01-12T01:02:25.921978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:26.219191+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:02:26.219191+0000","last_peered":"2024-01-12T01:02:26.219191+0000","last_clean":"2024-01-12T01:02:26.219191+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:02:26.219191+0000","last_undegraded":"2024-01-12T01:02:26.219191+0000","last_fullsized":"2024-01-12T01:02:26.219191+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740782,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.47399999999999998,"5min":0.41299999999999998,"15min":0.33100000000000002},"max":{"1min":0.71199999999999997,"5min":0.754,"15min":1.028},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Jan 12 01:03:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.58099999999999996},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":0.79600000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.504},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67600000000000005,"15min":0.68500000000000005},"min":{"1min":0.59099999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79200000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.64900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740780,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.44800000000000001,"5min":0.40799999999999997,"15min":0.35099999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.78300000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.54100000000000004,"5min":0.51200000000000001,"15min":0.39700000000000002},"max":{"1min":0.81399999999999995,"5min":0.875,"15min":0.875},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Jan 12 01:03:56 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61699999999999999,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.746,"5min":0.877,"15min":0.877},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.67600000000000005},"min":{"1min":0.65400000000000003,"5min":0.55900000000000005,"15min":0.28299999999999997},"max":{"1min":0.80100000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.69899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359740783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:04:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53800000000000003,"15min":0.54100000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.63300000000000001,"5min":0.751,"15min":0.79000000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.58899999999999997},"min":{"1min":0.46100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70699999999999996,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.64100000000000001}]},{"osd":2,"last update":"Fri Jan 12 01:04:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.67900000000000005,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65300000000000002,"15min":0.64200000000000002},"min":{"1min":0.502,"5min":0.48499999999999999,"15min":0.46500000000000002},"max":{"1min":0.76400000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.72599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:04:51.329 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.329 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.329 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.330 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.330 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.330 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.330 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.331 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.331 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.331 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:04:51.331 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:04:51.740 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:51.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:52.062 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:52.059+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:52.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:52.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:53.014 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:53.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:53.714 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:53.711+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:53.982 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:53.979+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:54.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:54.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:54.993 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:54.991+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:55.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:55.727+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:56.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:56.035+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:56.699 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:56.699+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:57.038 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:57.035+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:57.708 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:57.707+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:58.084 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:58.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:58.696 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:58.695+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:59.074 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:04:59.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:04:59.716 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:04:59.715+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:00.041 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:00.039+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:00.739 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:00.739+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:01.012 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:01.011+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:01.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:01.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:02.063 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:02.063+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:02.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:02.827+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:03.108 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:03.107+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:03.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:03.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:04.107 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:04.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:04.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:04.811+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:05.115 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:05.111+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:05.850 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:05.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:06.099 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:06.099+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:06.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:06.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:07.073 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:07.071+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:07.803 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:07.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:08.118 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:08.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:08.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:08.839+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:09.079 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:09.075+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:09.846 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:09.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:10.105 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:10.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:10.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:10.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:11.086 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:11.083+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:11.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:05:11.626 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:05:11.629 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:05:11.639 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6188,"stamp":"2024-01-12T01:05:09.913839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1980,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.503197+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:02:23.503197+0000","last_peered":"2024-01-12T01:02:23.503197+0000","last_clean":"2024-01-12T01:02:23.503197+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:02:23.503197+0000","last_undegraded":"2024-01-12T01:02:23.503197+0000","last_fullsized":"2024-01-12T01:02:23.503197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1713,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.805441+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:02:23.805441+0000","last_peered":"2024-01-12T01:02:23.805441+0000","last_clean":"2024-01-12T01:02:23.805441+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:02:23.805441+0000","last_undegraded":"2024-01-12T01:02:23.805441+0000","last_fullsized":"2024-01-12T01:02:23.805441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1925,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.105419+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:02:24.105419+0000","last_peered":"2024-01-12T01:02:24.105419+0000","last_clean":"2024-01-12T01:02:24.105419+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:02:24.105419+0000","last_undegraded":"2024-01-12T01:02:24.105419+0000","last_fullsized":"2024-01-12T01:02:24.105419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12871,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.407169+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:02:24.407169+0000","last_peered":"2024-01-12T01:02:24.407169+0000","last_clean":"2024-01-12T01:02:24.407169+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:02:24.407169+0000","last_undegraded":"2024-01-12T01:02:24.407169+0000","last_fullsized":"2024-01-12T01:02:24.407169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2144,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:02:24.701024+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:02:24.701024+0000","last_peered":"2024-01-12T01:02:24.701024+0000","last_clean":"2024-01-12T01:02:24.701024+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:02:24.701024+0000","last_undegraded":"2024-01-12T01:02:24.701024+0000","last_fullsized":"2024-01-12T01:02:24.701024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12355,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.000209+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:02:25.000209+0000","last_peered":"2024-01-12T01:02:25.000209+0000","last_clean":"2024-01-12T01:02:25.000209+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:02:25.000209+0000","last_undegraded":"2024-01-12T01:02:25.000209+0000","last_fullsized":"2024-01-12T01:02:25.000209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7506,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.285268+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:02:25.285268+0000","last_peered":"2024-01-12T01:02:25.285268+0000","last_clean":"2024-01-12T01:02:25.285268+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:02:25.285268+0000","last_undegraded":"2024-01-12T01:02:25.285268+0000","last_fullsized":"2024-01-12T01:02:25.285268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11783,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.586098+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:02:25.586098+0000","last_peered":"2024-01-12T01:02:25.586098+0000","last_clean":"2024-01-12T01:02:25.586098+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:02:25.586098+0000","last_undegraded":"2024-01-12T01:02:25.586098+0000","last_fullsized":"2024-01-12T01:02:25.586098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2521,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.921978+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:02:25.921978+0000","last_peered":"2024-01-12T01:02:25.921978+0000","last_clean":"2024-01-12T01:02:25.921978+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:02:25.921978+0000","last_undegraded":"2024-01-12T01:02:25.921978+0000","last_fullsized":"2024-01-12T01:02:25.921978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:26.219191+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:02:26.219191+0000","last_peered":"2024-01-12T01:02:26.219191+0000","last_clean":"2024-01-12T01:02:26.219191+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:02:26.219191+0000","last_undegraded":"2024-01-12T01:02:26.219191+0000","last_fullsized":"2024-01-12T01:02:26.219191+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740786,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:04:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.47399999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58699999999999997,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.69999999999999996,"5min":0.754,"15min":1.028},"last":0.68700000000000006}]},{"osd":1,"last update":"Fri Jan 12 01:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.48699999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.69299999999999995,"5min":0.79600000000000004,"15min":0.80300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.68300000000000005},"min":{"1min":0.52000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78400000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.77400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740784,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.44800000000000001,"5min":0.40799999999999997,"15min":0.35099999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.78300000000000003},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63300000000000001,"15min":0.622},"min":{"1min":0.54100000000000004,"5min":0.51200000000000001,"15min":0.39700000000000002},"max":{"1min":0.81399999999999995,"5min":0.875,"15min":0.875},"last":0.64000000000000001}]},{"osd":2,"last update":"Fri Jan 12 01:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.59399999999999997},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.753,"5min":0.877,"15min":0.877},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.67600000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.82199999999999995,"5min":0.93500000000000005,"15min":1.071},"last":0.69599999999999995}]}]},{"osd":0,"up_from":8,"seq":34359740787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:04:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53800000000000003,"15min":0.54100000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.63300000000000001,"5min":0.751,"15min":0.79000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.58899999999999997},"min":{"1min":0.46100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70699999999999996,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:04:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.67900000000000005,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65300000000000002,"15min":0.64200000000000002},"min":{"1min":0.502,"5min":0.48499999999999999,"15min":0.46500000000000002},"max":{"1min":0.76400000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:05:11.640 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.640 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.640 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.640 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.641 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.641 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.641 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.641 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.641 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.641 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:11.641 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:05:11.867 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:11.867+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:12.106 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:12.103+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:12.873 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:12.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:13.116 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:13.115+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:13.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:13.843+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:14.151 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:14.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:14.876 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:14.875+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:15.172 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:15.171+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:15.864 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:15.863+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:16.154 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:16.151+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:16.859 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:16.859+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:17.194 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:17.191+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:17.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:17.874+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:18.212 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:18.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:18.897 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:18.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:19.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:19.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:19.913 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:19.910+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:20.179 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:20.178+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:20.960 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:20.958+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:21.207 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:21.206+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:21.954 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:21.950+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:22.169 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:22.166+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:22.926 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:22.922+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:23.136 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:23.134+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:23.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:23.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:24.148 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:24.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:24.847 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:24.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:25.130 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:25.126+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:25.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:25.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:26.094 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:26.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:26.892 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:26.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:27.048 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:27.046+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:27.869 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:27.866+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:28.091 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:28.090+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:28.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:28.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:29.120 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:29.118+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:29.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:29.846+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:30.097 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:30.094+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:30.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:30.890+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:31.117 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:31.114+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:31.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:05:31.898 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:31.894+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:31.930 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:05:31.933 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:05:31.943 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6199,"stamp":"2024-01-12T01:05:31.919818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1980,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.503197+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:02:23.503197+0000","last_peered":"2024-01-12T01:02:23.503197+0000","last_clean":"2024-01-12T01:02:23.503197+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:02:23.503197+0000","last_undegraded":"2024-01-12T01:02:23.503197+0000","last_fullsized":"2024-01-12T01:02:23.503197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1713,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.805441+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:02:23.805441+0000","last_peered":"2024-01-12T01:02:23.805441+0000","last_clean":"2024-01-12T01:02:23.805441+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:02:23.805441+0000","last_undegraded":"2024-01-12T01:02:23.805441+0000","last_fullsized":"2024-01-12T01:02:23.805441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1925,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.105419+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:02:24.105419+0000","last_peered":"2024-01-12T01:02:24.105419+0000","last_clean":"2024-01-12T01:02:24.105419+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:02:24.105419+0000","last_undegraded":"2024-01-12T01:02:24.105419+0000","last_fullsized":"2024-01-12T01:02:24.105419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12871,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.407169+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:02:24.407169+0000","last_peered":"2024-01-12T01:02:24.407169+0000","last_clean":"2024-01-12T01:02:24.407169+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:02:24.407169+0000","last_undegraded":"2024-01-12T01:02:24.407169+0000","last_fullsized":"2024-01-12T01:02:24.407169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2144,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:02:24.701024+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:02:24.701024+0000","last_peered":"2024-01-12T01:02:24.701024+0000","last_clean":"2024-01-12T01:02:24.701024+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:02:24.701024+0000","last_undegraded":"2024-01-12T01:02:24.701024+0000","last_fullsized":"2024-01-12T01:02:24.701024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12355,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.000209+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:02:25.000209+0000","last_peered":"2024-01-12T01:02:25.000209+0000","last_clean":"2024-01-12T01:02:25.000209+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:02:25.000209+0000","last_undegraded":"2024-01-12T01:02:25.000209+0000","last_fullsized":"2024-01-12T01:02:25.000209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7506,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.285268+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:02:25.285268+0000","last_peered":"2024-01-12T01:02:25.285268+0000","last_clean":"2024-01-12T01:02:25.285268+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:02:25.285268+0000","last_undegraded":"2024-01-12T01:02:25.285268+0000","last_fullsized":"2024-01-12T01:02:25.285268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11783,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.586098+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:02:25.586098+0000","last_peered":"2024-01-12T01:02:25.586098+0000","last_clean":"2024-01-12T01:02:25.586098+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:02:25.586098+0000","last_undegraded":"2024-01-12T01:02:25.586098+0000","last_fullsized":"2024-01-12T01:02:25.586098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2521,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.921978+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:02:25.921978+0000","last_peered":"2024-01-12T01:02:25.921978+0000","last_clean":"2024-01-12T01:02:25.921978+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:02:25.921978+0000","last_undegraded":"2024-01-12T01:02:25.921978+0000","last_fullsized":"2024-01-12T01:02:25.921978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:26.219191+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:02:26.219191+0000","last_peered":"2024-01-12T01:02:26.219191+0000","last_clean":"2024-01-12T01:02:26.219191+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:02:26.219191+0000","last_undegraded":"2024-01-12T01:02:26.219191+0000","last_fullsized":"2024-01-12T01:02:26.219191+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740791,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:04:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.47399999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.62},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58699999999999997,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.69999999999999996,"5min":0.754,"15min":1.028},"last":0.68899999999999995}]},{"osd":1,"last update":"Fri Jan 12 01:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.48699999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.69299999999999995,"5min":0.79600000000000004,"15min":0.80300000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.68300000000000005},"min":{"1min":0.52000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78400000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.79200000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740789,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.71099999999999997,"15min":0.78300000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.626,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.78900000000000003,"5min":0.875,"15min":0.875},"last":0.68000000000000005}]},{"osd":2,"last update":"Fri Jan 12 01:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.59399999999999997},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.753,"5min":0.877,"15min":0.877},"last":0.628},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.67600000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.82199999999999995,"5min":0.93500000000000005,"15min":1.071},"last":0.746}]}]},{"osd":0,"up_from":8,"seq":34359740792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:04:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53800000000000003,"15min":0.54100000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.63300000000000001,"5min":0.751,"15min":0.79000000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.58899999999999997},"min":{"1min":0.46100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70699999999999996,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.59999999999999998}]},{"osd":2,"last update":"Fri Jan 12 01:04:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.67900000000000005,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65300000000000002,"15min":0.64200000000000002},"min":{"1min":0.502,"5min":0.48499999999999999,"15min":0.46500000000000002},"max":{"1min":0.76400000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:05:31.944 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.944 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.944 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.945 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.945 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.945 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.945 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.945 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.945 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.946 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:31.946 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:05:32.101 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:32.098+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:32.885 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:32.882+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:33.150 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:33.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:33.904 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:33.902+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:34.166 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:34.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:34.874 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:34.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:35.214 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:35.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:35.866 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:35.862+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:36.245 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:36.242+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:36.872 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:36.870+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:37.200 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:37.198+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:37.860 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:37.858+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:38.221 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:38.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:38.820 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:38.818+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:39.184 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:39.182+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:39.842 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:39.838+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:40.153 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:40.150+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:40.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:40.822+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:41.163 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:41.162+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:41.795 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:41.794+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:42.149 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:42.146+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:42.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:42.750+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:43.197 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:43.194+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:43.749 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:43.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:44.218 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:44.214+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:44.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:44.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:45.264 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:45.262+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:45.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:45.786+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:46.222 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:46.218+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:46.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:46.758+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:47.213 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:47.210+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:47.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:47.746+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:48.252 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:48.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:48.782 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:48.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:49.259 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:49.254+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:49.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:49.742+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:50.251 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:50.250+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:50.781 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:50.778+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:51.277 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:51.274+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:51.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:51.762+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:51.947 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:05:52.238 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:05:52.241 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:05:52.250 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6209,"stamp":"2024-01-12T01:05:51.925055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003114"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1980,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.503197+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:02:23.503197+0000","last_peered":"2024-01-12T01:02:23.503197+0000","last_clean":"2024-01-12T01:02:23.503197+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:02:23.503197+0000","last_undegraded":"2024-01-12T01:02:23.503197+0000","last_fullsized":"2024-01-12T01:02:23.503197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1713,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.805441+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:02:23.805441+0000","last_peered":"2024-01-12T01:02:23.805441+0000","last_clean":"2024-01-12T01:02:23.805441+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:02:23.805441+0000","last_undegraded":"2024-01-12T01:02:23.805441+0000","last_fullsized":"2024-01-12T01:02:23.805441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1925,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.105419+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:02:24.105419+0000","last_peered":"2024-01-12T01:02:24.105419+0000","last_clean":"2024-01-12T01:02:24.105419+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:02:24.105419+0000","last_undegraded":"2024-01-12T01:02:24.105419+0000","last_fullsized":"2024-01-12T01:02:24.105419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12871,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.407169+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:02:24.407169+0000","last_peered":"2024-01-12T01:02:24.407169+0000","last_clean":"2024-01-12T01:02:24.407169+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:02:24.407169+0000","last_undegraded":"2024-01-12T01:02:24.407169+0000","last_fullsized":"2024-01-12T01:02:24.407169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2144,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:02:24.701024+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:02:24.701024+0000","last_peered":"2024-01-12T01:02:24.701024+0000","last_clean":"2024-01-12T01:02:24.701024+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:02:24.701024+0000","last_undegraded":"2024-01-12T01:02:24.701024+0000","last_fullsized":"2024-01-12T01:02:24.701024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12355,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.000209+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:02:25.000209+0000","last_peered":"2024-01-12T01:02:25.000209+0000","last_clean":"2024-01-12T01:02:25.000209+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:02:25.000209+0000","last_undegraded":"2024-01-12T01:02:25.000209+0000","last_fullsized":"2024-01-12T01:02:25.000209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7506,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.285268+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:02:25.285268+0000","last_peered":"2024-01-12T01:02:25.285268+0000","last_clean":"2024-01-12T01:02:25.285268+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:02:25.285268+0000","last_undegraded":"2024-01-12T01:02:25.285268+0000","last_fullsized":"2024-01-12T01:02:25.285268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11783,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.586098+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:02:25.586098+0000","last_peered":"2024-01-12T01:02:25.586098+0000","last_clean":"2024-01-12T01:02:25.586098+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:02:25.586098+0000","last_undegraded":"2024-01-12T01:02:25.586098+0000","last_fullsized":"2024-01-12T01:02:25.586098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2521,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.921978+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:02:25.921978+0000","last_peered":"2024-01-12T01:02:25.921978+0000","last_clean":"2024-01-12T01:02:25.921978+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:02:25.921978+0000","last_undegraded":"2024-01-12T01:02:25.921978+0000","last_fullsized":"2024-01-12T01:02:25.921978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:26.219191+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:02:26.219191+0000","last_peered":"2024-01-12T01:02:26.219191+0000","last_clean":"2024-01-12T01:02:26.219191+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:02:26.219191+0000","last_undegraded":"2024-01-12T01:02:26.219191+0000","last_fullsized":"2024-01-12T01:02:26.219191+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740795,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:04:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.47399999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58699999999999997,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.69999999999999996,"5min":0.754,"15min":1.028},"last":0.70999999999999996}]},{"osd":1,"last update":"Fri Jan 12 01:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.48699999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.69299999999999995,"5min":0.79600000000000004,"15min":0.80300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.68300000000000005},"min":{"1min":0.52000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78400000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740793,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.71099999999999997,"15min":0.78300000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.626,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.78900000000000003,"5min":0.875,"15min":0.875},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Jan 12 01:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.59399999999999997},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.753,"5min":0.877,"15min":0.877},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.67600000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.82199999999999995,"5min":0.93500000000000005,"15min":1.071},"last":0.76300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.41699999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.622,"5min":0.751,"15min":0.79000000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80700000000000005,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.65000000000000002}]},{"osd":2,"last update":"Fri Jan 12 01:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58899999999999997,"15min":0.59999999999999998},"min":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.71799999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66100000000000003,"15min":0.64300000000000002},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.46500000000000002},"max":{"1min":0.74399999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:05:52.252 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.252 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.252 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.253 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.253 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.253 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.253 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.253 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.253 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.254 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:05:52.254 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:05:52.312 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:52.310+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:52.774 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:52.770+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:53.331 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:53.330+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:53.729 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:53.726+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:54.359 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:54.358+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:54.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:54.698+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:55.392 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:55.390+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:55.705 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:55.702+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:56.413 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:56.410+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:56.663 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:56.662+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:57.446 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:57.442+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:57.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:57.626+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:58.453 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:58.450+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:58.596 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:58.594+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:59.415 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:05:59.414+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:05:59.603 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:05:59.598+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:00.417 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:00.414+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:00.638 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:00.634+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:01.448 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:01.446+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:01.632 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:01.630+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:02.420 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:02.418+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:02.612 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:02.610+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:03.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:03.462+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:03.589 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:03.586+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:04.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:04.486+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:04.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:04.618+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:05.490 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:05.486+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:05.610 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:05.606+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:06.481 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:06.478+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:06.627 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:06.626+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:07.488 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:07.486+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:07.645 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:07.642+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:08.487 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:08.486+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:08.629 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:08.626+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:09.465 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:09.462+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:09.613 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:09.610+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:10.484 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:10.481+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:10.598 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:10.593+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:11.527 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:11.525+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:11.583 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:11.581+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:12.255 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:06:12.523 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:12.521+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:12.548 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:06:12.551 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:06:12.561 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6219,"stamp":"2024-01-12T01:06:11.930515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003099"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1980,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.503197+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:02:23.503197+0000","last_peered":"2024-01-12T01:02:23.503197+0000","last_clean":"2024-01-12T01:02:23.503197+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:02:23.503197+0000","last_undegraded":"2024-01-12T01:02:23.503197+0000","last_fullsized":"2024-01-12T01:02:23.503197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1713,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.805441+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:02:23.805441+0000","last_peered":"2024-01-12T01:02:23.805441+0000","last_clean":"2024-01-12T01:02:23.805441+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:02:23.805441+0000","last_undegraded":"2024-01-12T01:02:23.805441+0000","last_fullsized":"2024-01-12T01:02:23.805441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1925,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.105419+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:02:24.105419+0000","last_peered":"2024-01-12T01:02:24.105419+0000","last_clean":"2024-01-12T01:02:24.105419+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:02:24.105419+0000","last_undegraded":"2024-01-12T01:02:24.105419+0000","last_fullsized":"2024-01-12T01:02:24.105419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12871,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.407169+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:02:24.407169+0000","last_peered":"2024-01-12T01:02:24.407169+0000","last_clean":"2024-01-12T01:02:24.407169+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:02:24.407169+0000","last_undegraded":"2024-01-12T01:02:24.407169+0000","last_fullsized":"2024-01-12T01:02:24.407169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2144,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:02:24.701024+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:02:24.701024+0000","last_peered":"2024-01-12T01:02:24.701024+0000","last_clean":"2024-01-12T01:02:24.701024+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:02:24.701024+0000","last_undegraded":"2024-01-12T01:02:24.701024+0000","last_fullsized":"2024-01-12T01:02:24.701024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12355,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.000209+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:02:25.000209+0000","last_peered":"2024-01-12T01:02:25.000209+0000","last_clean":"2024-01-12T01:02:25.000209+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:02:25.000209+0000","last_undegraded":"2024-01-12T01:02:25.000209+0000","last_fullsized":"2024-01-12T01:02:25.000209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7506,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.285268+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:02:25.285268+0000","last_peered":"2024-01-12T01:02:25.285268+0000","last_clean":"2024-01-12T01:02:25.285268+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:02:25.285268+0000","last_undegraded":"2024-01-12T01:02:25.285268+0000","last_fullsized":"2024-01-12T01:02:25.285268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11783,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.586098+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:02:25.586098+0000","last_peered":"2024-01-12T01:02:25.586098+0000","last_clean":"2024-01-12T01:02:25.586098+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:02:25.586098+0000","last_undegraded":"2024-01-12T01:02:25.586098+0000","last_fullsized":"2024-01-12T01:02:25.586098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2521,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.921978+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:02:25.921978+0000","last_peered":"2024-01-12T01:02:25.921978+0000","last_clean":"2024-01-12T01:02:25.921978+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:02:25.921978+0000","last_undegraded":"2024-01-12T01:02:25.921978+0000","last_fullsized":"2024-01-12T01:02:25.921978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:26.219191+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:02:26.219191+0000","last_peered":"2024-01-12T01:02:26.219191+0000","last_clean":"2024-01-12T01:02:26.219191+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:02:26.219191+0000","last_undegraded":"2024-01-12T01:02:26.219191+0000","last_fullsized":"2024-01-12T01:02:26.219191+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740799,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:05:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.45200000000000001,"5min":0.432,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.622},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.57699999999999996},"min":{"1min":0.498,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.754,"15min":1.028},"last":0.56799999999999995}]},{"osd":1,"last update":"Fri Jan 12 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.374},"max":{"1min":0.65300000000000002,"5min":0.79600000000000004,"15min":0.80300000000000005},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.42299999999999999},"max":{"1min":0.86099999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740797,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.71099999999999997,"15min":0.78300000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.626,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.78900000000000003,"5min":0.875,"15min":0.875},"last":0.56200000000000006}]},{"osd":2,"last update":"Fri Jan 12 01:05:59 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61699999999999999,"15min":0.59799999999999998},"min":{"1min":0.54400000000000004,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.73799999999999999,"5min":0.877,"15min":0.877},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.67900000000000005},"min":{"1min":0.63500000000000001,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.86899999999999999,"5min":0.93500000000000005,"15min":1.071},"last":0.68700000000000006}]}]},{"osd":0,"up_from":8,"seq":34359740800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.41699999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.622,"5min":0.751,"15min":0.79000000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80700000000000005,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.58099999999999996}]},{"osd":2,"last update":"Fri Jan 12 01:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58899999999999997,"15min":0.59999999999999998},"min":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.71799999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66100000000000003,"15min":0.64300000000000002},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.46500000000000002},"max":{"1min":0.74399999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:06:12.563 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.563 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.563 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.563 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.564 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.564 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.564 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.564 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.565 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.565 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:12.565 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:06:12.625 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:12.621+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:13.567 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:13.565+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:13.662 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:13.657+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:14.561 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:14.557+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:14.654 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:14.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:15.535 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:15.533+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:15.686 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:15.681+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:16.507 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:16.501+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:16.728 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:16.725+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:17.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:17.549+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:17.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:17.709+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:18.583 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:18.581+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:18.687 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:18.685+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:19.626 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:19.621+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:19.648 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:19.645+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:20.628 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:20.625+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:20.669 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:20.665+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:21.621 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:21.617+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:21.634 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:21.629+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:22.590 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:22.585+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:22.630 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:22.625+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:23.603 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:23.601+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:23.607 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:23.605+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:24.577 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:24.573+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:24.652 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:24.649+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:25.574 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:25.569+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:25.628 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:25.625+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:26.550 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:26.545+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:26.671 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:26.669+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:27.551 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:27.549+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:27.704 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:27.701+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:28.592 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:28.589+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:28.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:28.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:29.604 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:29.601+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:29.746 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:29.741+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:30.560 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:30.557+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:30.773 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:30.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:31.575 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:31.573+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:31.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:31.805+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:32.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:06:32.621 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:32.617+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:32.816 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:32.813+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:32.860 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:06:32.863 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:06:32.873 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6229,"stamp":"2024-01-12T01:06:31.936331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003465"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1980,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.503197+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:02:23.503197+0000","last_peered":"2024-01-12T01:02:23.503197+0000","last_clean":"2024-01-12T01:02:23.503197+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:02:23.503197+0000","last_undegraded":"2024-01-12T01:02:23.503197+0000","last_fullsized":"2024-01-12T01:02:23.503197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1713,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:23.805441+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:02:23.805441+0000","last_peered":"2024-01-12T01:02:23.805441+0000","last_clean":"2024-01-12T01:02:23.805441+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:02:23.805441+0000","last_undegraded":"2024-01-12T01:02:23.805441+0000","last_fullsized":"2024-01-12T01:02:23.805441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1925,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.105419+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:02:24.105419+0000","last_peered":"2024-01-12T01:02:24.105419+0000","last_clean":"2024-01-12T01:02:24.105419+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:02:24.105419+0000","last_undegraded":"2024-01-12T01:02:24.105419+0000","last_fullsized":"2024-01-12T01:02:24.105419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12871,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:24.407169+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:02:24.407169+0000","last_peered":"2024-01-12T01:02:24.407169+0000","last_clean":"2024-01-12T01:02:24.407169+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:02:24.407169+0000","last_undegraded":"2024-01-12T01:02:24.407169+0000","last_fullsized":"2024-01-12T01:02:24.407169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2144,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:02:24.701024+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:02:24.701024+0000","last_peered":"2024-01-12T01:02:24.701024+0000","last_clean":"2024-01-12T01:02:24.701024+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:02:24.701024+0000","last_undegraded":"2024-01-12T01:02:24.701024+0000","last_fullsized":"2024-01-12T01:02:24.701024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12355,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.000209+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:02:25.000209+0000","last_peered":"2024-01-12T01:02:25.000209+0000","last_clean":"2024-01-12T01:02:25.000209+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:02:25.000209+0000","last_undegraded":"2024-01-12T01:02:25.000209+0000","last_fullsized":"2024-01-12T01:02:25.000209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7506,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.285268+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:02:25.285268+0000","last_peered":"2024-01-12T01:02:25.285268+0000","last_clean":"2024-01-12T01:02:25.285268+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:02:25.285268+0000","last_undegraded":"2024-01-12T01:02:25.285268+0000","last_fullsized":"2024-01-12T01:02:25.285268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11783,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.586098+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:02:25.586098+0000","last_peered":"2024-01-12T01:02:25.586098+0000","last_clean":"2024-01-12T01:02:25.586098+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:02:25.586098+0000","last_undegraded":"2024-01-12T01:02:25.586098+0000","last_fullsized":"2024-01-12T01:02:25.586098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2521,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:25.921978+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:02:25.921978+0000","last_peered":"2024-01-12T01:02:25.921978+0000","last_clean":"2024-01-12T01:02:25.921978+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:02:25.921978+0000","last_undegraded":"2024-01-12T01:02:25.921978+0000","last_fullsized":"2024-01-12T01:02:25.921978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:02:26.219191+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:02:26.219191+0000","last_peered":"2024-01-12T01:02:26.219191+0000","last_clean":"2024-01-12T01:02:26.219191+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:02:26.219191+0000","last_undegraded":"2024-01-12T01:02:26.219191+0000","last_fullsized":"2024-01-12T01:02:26.219191+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740803,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:05:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.45200000000000001,"5min":0.432,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.57699999999999996},"min":{"1min":0.498,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.754,"15min":1.028},"last":0.66500000000000004}]},{"osd":1,"last update":"Fri Jan 12 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.374},"max":{"1min":0.65300000000000002,"5min":0.79600000000000004,"15min":0.80300000000000005},"last":0.629},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.42299999999999999},"max":{"1min":0.86099999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359740801,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.40400000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.77900000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.622,"15min":0.621},"min":{"1min":0.51300000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.74299999999999999,"5min":0.875,"15min":0.875},"last":0.64000000000000001}]},{"osd":2,"last update":"Fri Jan 12 01:05:59 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61699999999999999,"15min":0.59799999999999998},"min":{"1min":0.54400000000000004,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.73799999999999999,"5min":0.877,"15min":0.877},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.67900000000000005},"min":{"1min":0.63500000000000001,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.86899999999999999,"5min":0.93500000000000005,"15min":1.071},"last":0.755}]}]},{"osd":0,"up_from":8,"seq":34359740804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.41699999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.622,"5min":0.751,"15min":0.79000000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80700000000000005,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.63500000000000001}]},{"osd":2,"last update":"Fri Jan 12 01:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58899999999999997,"15min":0.59999999999999998},"min":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.71799999999999997,"5min":0.78300000000000003,"15min":0.81599999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66100000000000003,"15min":0.64300000000000002},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.46500000000000002},"max":{"1min":0.74399999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.745}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:06:32.874 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.875 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.875 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.875 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.875 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.875 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.876 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.876 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.876 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.876 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:32.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2024-01-12T01:06:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2024-01-12T01:06:33.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2024-01-12T01:06:33.468 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2024-01-12T01:06:33.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2024-01-12T01:06:33.648 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:33.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:33.759 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2024-01-12T01:06:33.769 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2024-01-12T01:06:33.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:33.777+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:34.056 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2024-01-12T01:06:34.065 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2024-01-12T01:06:34.406 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2024-01-12T01:06:34.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2024-01-12T01:06:34.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:34.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:34.715 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-01-12T01:06:34.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2024-01-12T01:06:34.764 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:34.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:35.210 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2024-01-12T01:06:35.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2024-01-12T01:06:35.555 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2024-01-12T01:06:35.565 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2024-01-12T01:06:35.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:35.665+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:35.724 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:35.721+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:35.850 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2024-01-12T01:06:35.860 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2024-01-12T01:06:36.145 INFO:teuthology.orchestra.run.smithi178.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-01-12T01:06:36.155 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:06:36.648 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:36.645+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:36.763 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:36.761+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:37.661 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:37.657+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:37.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:37.769+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:38.614 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:38.609+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:38.784 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:38.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:39.595 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:39.593+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:39.785 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:39.781+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:40.620 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:40.617+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:40.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:40.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:41.601 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:41.597+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:41.829 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:41.825+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:42.629 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:42.625+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:42.853 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:42.849+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:43.640 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:43.637+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:43.893 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:43.889+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:44.686 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:44.681+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:44.939 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:44.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:45.672 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:45.669+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:45.958 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:45.953+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:46.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:46.701+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:46.947 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:46.941+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:47.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:47.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:47.915 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:47.913+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:48.740 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:48.737+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:48.920 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:48.917+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:49.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:49.761+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:49.899 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:49.897+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:50.735 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:50.729+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:50.949 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:50.945+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:51.712 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:51.709+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:51.934 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:51.929+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:52.749 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:52.745+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:52.980 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:52.977+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:53.787 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:53.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:53.942 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:53.937+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:54.791 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:54.785+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:54.896 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:54.893+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:55.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:55.793+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:55.928 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:55.925+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:56.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:06:56.453 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:06:56.456 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:06:56.470 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6241,"stamp":"2024-01-12T01:06:55.942777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003286"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1981,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.357802+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:04:28.357802+0000","last_peered":"2024-01-12T01:04:28.357802+0000","last_clean":"2024-01-12T01:04:28.357802+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:04:28.357802+0000","last_undegraded":"2024-01-12T01:04:28.357802+0000","last_fullsized":"2024-01-12T01:04:28.357802+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1714,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.650810+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:04:28.650810+0000","last_peered":"2024-01-12T01:04:28.650810+0000","last_clean":"2024-01-12T01:04:28.650810+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:04:28.650810+0000","last_undegraded":"2024-01-12T01:04:28.650810+0000","last_fullsized":"2024-01-12T01:04:28.650810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1926,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.942334+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:04:28.942334+0000","last_peered":"2024-01-12T01:04:28.942334+0000","last_clean":"2024-01-12T01:04:28.942334+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:04:28.942334+0000","last_undegraded":"2024-01-12T01:04:28.942334+0000","last_fullsized":"2024-01-12T01:04:28.942334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12872,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.235425+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:04:29.235425+0000","last_peered":"2024-01-12T01:04:29.235425+0000","last_clean":"2024-01-12T01:04:29.235425+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:04:29.235425+0000","last_undegraded":"2024-01-12T01:04:29.235425+0000","last_fullsized":"2024-01-12T01:04:29.235425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2145,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:04:29.529311+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:04:29.529311+0000","last_peered":"2024-01-12T01:04:29.529311+0000","last_clean":"2024-01-12T01:04:29.529311+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:04:29.529311+0000","last_undegraded":"2024-01-12T01:04:29.529311+0000","last_fullsized":"2024-01-12T01:04:29.529311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 765s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12356,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.818668+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:04:29.818668+0000","last_peered":"2024-01-12T01:04:29.818668+0000","last_clean":"2024-01-12T01:04:29.818668+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:04:29.818668+0000","last_undegraded":"2024-01-12T01:04:29.818668+0000","last_fullsized":"2024-01-12T01:04:29.818668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7507,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.127871+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:04:30.127871+0000","last_peered":"2024-01-12T01:04:30.127871+0000","last_clean":"2024-01-12T01:04:30.127871+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:04:30.127871+0000","last_undegraded":"2024-01-12T01:04:30.127871+0000","last_fullsized":"2024-01-12T01:04:30.127871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11784,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.423668+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:04:30.423668+0000","last_peered":"2024-01-12T01:04:30.423668+0000","last_clean":"2024-01-12T01:04:30.423668+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:04:30.423668+0000","last_undegraded":"2024-01-12T01:04:30.423668+0000","last_fullsized":"2024-01-12T01:04:30.423668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2522,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.722844+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:04:30.722844+0000","last_peered":"2024-01-12T01:04:30.722844+0000","last_clean":"2024-01-12T01:04:30.722844+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:04:30.722844+0000","last_undegraded":"2024-01-12T01:04:30.722844+0000","last_fullsized":"2024-01-12T01:04:30.722844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:31.013211+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:04:31.013211+0000","last_peered":"2024-01-12T01:04:31.013211+0000","last_clean":"2024-01-12T01:04:31.013211+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:04:31.013211+0000","last_undegraded":"2024-01-12T01:04:31.013211+0000","last_fullsized":"2024-01-12T01:04:31.013211+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740808,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:05:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.45200000000000001,"5min":0.432,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.57699999999999996},"min":{"1min":0.498,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.754,"15min":1.028},"last":0.67300000000000004}]},{"osd":1,"last update":"Fri Jan 12 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.374},"max":{"1min":0.65300000000000002,"5min":0.79600000000000004,"15min":0.80300000000000005},"last":0.626},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.42299999999999999},"max":{"1min":0.86099999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.57499999999999996}]}]},{"osd":1,"up_from":8,"seq":34359740806,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.40400000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.77900000000000003},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.622,"15min":0.621},"min":{"1min":0.51300000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.74299999999999999,"5min":0.875,"15min":0.875},"last":0.72699999999999998}]},{"osd":2,"last update":"Fri Jan 12 01:05:59 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61699999999999999,"15min":0.59799999999999998},"min":{"1min":0.54400000000000004,"5min":0.44600000000000001,"15min":0.439},"max":{"1min":0.73799999999999999,"5min":0.877,"15min":0.877},"last":0.626},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.67900000000000005},"min":{"1min":0.63500000000000001,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.86899999999999999,"5min":0.93500000000000005,"15min":1.071},"last":0.81100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359740809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.65200000000000002,"5min":0.751,"15min":0.79000000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.48499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72299999999999998,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.60099999999999998},"min":{"1min":0.47999999999999998,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.70699999999999996,"5min":0.71999999999999997,"15min":0.81599999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.745,"5min":0.76400000000000001,"15min":0.82999999999999996},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:06:56.471 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.472 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.472 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.472 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.472 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.472 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.473 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.473 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.473 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.473 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:06:56.473 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:06:56.776 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:56.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:56.973 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:56.969+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:57.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:57.773+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:57.951 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:57.949+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:58.774 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:58.769+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:58.925 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:58.921+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:59.753 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:06:59.749+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:06:59.888 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:06:59.885+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:00.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:00.749+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:00.844 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:00.841+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:01.728 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:01.725+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:01.871 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:01.869+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:02.737 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:02.732+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:02.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:02.828+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:03.768 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:03.764+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:03.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:03.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:04.754 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:04.748+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:04.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:04.784+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:05.717 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:05.712+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:05.766 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:05.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:06.751 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:06.744+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:06.798 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:06.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:07.748 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:07.744+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:07.793 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:07.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:08.779 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:08.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:08.798 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:08.792+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:09.811 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:09.808+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:09.813 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:09.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:10.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:10.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:10.808 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:10.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:11.772 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:11.768+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:11.791 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:11.788+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:12.800 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:12.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:12.811 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:12.804+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:13.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:13.764+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:13.804 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:13.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:14.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:14.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:14.793 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:14.788+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:15.765 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:15.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:15.818 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:15.812+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:16.474 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:07:16.773 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:07:16.776 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:07:16.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:16.776+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:16.785 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:16.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:16.792 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6251,"stamp":"2024-01-12T01:07:15.948279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003153"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1981,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.357802+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:04:28.357802+0000","last_peered":"2024-01-12T01:04:28.357802+0000","last_clean":"2024-01-12T01:04:28.357802+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:04:28.357802+0000","last_undegraded":"2024-01-12T01:04:28.357802+0000","last_fullsized":"2024-01-12T01:04:28.357802+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1714,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.650810+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:04:28.650810+0000","last_peered":"2024-01-12T01:04:28.650810+0000","last_clean":"2024-01-12T01:04:28.650810+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:04:28.650810+0000","last_undegraded":"2024-01-12T01:04:28.650810+0000","last_fullsized":"2024-01-12T01:04:28.650810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1926,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.942334+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:04:28.942334+0000","last_peered":"2024-01-12T01:04:28.942334+0000","last_clean":"2024-01-12T01:04:28.942334+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:04:28.942334+0000","last_undegraded":"2024-01-12T01:04:28.942334+0000","last_fullsized":"2024-01-12T01:04:28.942334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12872,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.235425+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:04:29.235425+0000","last_peered":"2024-01-12T01:04:29.235425+0000","last_clean":"2024-01-12T01:04:29.235425+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:04:29.235425+0000","last_undegraded":"2024-01-12T01:04:29.235425+0000","last_fullsized":"2024-01-12T01:04:29.235425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2145,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:04:29.529311+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:04:29.529311+0000","last_peered":"2024-01-12T01:04:29.529311+0000","last_clean":"2024-01-12T01:04:29.529311+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:04:29.529311+0000","last_undegraded":"2024-01-12T01:04:29.529311+0000","last_fullsized":"2024-01-12T01:04:29.529311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 765s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12356,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.818668+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:04:29.818668+0000","last_peered":"2024-01-12T01:04:29.818668+0000","last_clean":"2024-01-12T01:04:29.818668+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:04:29.818668+0000","last_undegraded":"2024-01-12T01:04:29.818668+0000","last_fullsized":"2024-01-12T01:04:29.818668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7507,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.127871+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:04:30.127871+0000","last_peered":"2024-01-12T01:04:30.127871+0000","last_clean":"2024-01-12T01:04:30.127871+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:04:30.127871+0000","last_undegraded":"2024-01-12T01:04:30.127871+0000","last_fullsized":"2024-01-12T01:04:30.127871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11784,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.423668+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:04:30.423668+0000","last_peered":"2024-01-12T01:04:30.423668+0000","last_clean":"2024-01-12T01:04:30.423668+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:04:30.423668+0000","last_undegraded":"2024-01-12T01:04:30.423668+0000","last_fullsized":"2024-01-12T01:04:30.423668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2522,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.722844+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:04:30.722844+0000","last_peered":"2024-01-12T01:04:30.722844+0000","last_clean":"2024-01-12T01:04:30.722844+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:04:30.722844+0000","last_undegraded":"2024-01-12T01:04:30.722844+0000","last_fullsized":"2024-01-12T01:04:30.722844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:31.013211+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:04:31.013211+0000","last_peered":"2024-01-12T01:04:31.013211+0000","last_clean":"2024-01-12T01:04:31.013211+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:04:31.013211+0000","last_undegraded":"2024-01-12T01:04:31.013211+0000","last_fullsized":"2024-01-12T01:04:31.013211+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740812,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54500000000000004,"15min":0.53600000000000003},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.32600000000000001},"max":{"1min":0.67000000000000004,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.45800000000000002,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.754,"15min":1.028},"last":0.58799999999999997}]},{"osd":1,"last update":"Fri Jan 12 01:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.58099999999999996},"min":{"1min":0.504,"5min":0.47099999999999997,"15min":0.374},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.86099999999999999,"15min":0.92700000000000005},"last":0.69599999999999995}]}]},{"osd":1,"up_from":8,"seq":34359740810,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.40400000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.77900000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.622,"15min":0.621},"min":{"1min":0.51300000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.74299999999999999,"5min":0.875,"15min":0.875},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Jan 12 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.626,"15min":0.59999999999999998},"min":{"1min":0.54600000000000004,"5min":0.46600000000000003,"15min":0.439},"max":{"1min":0.752,"5min":0.877,"15min":0.877},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.71199999999999997,"15min":0.68100000000000005},"min":{"1min":0.626,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.81100000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.72399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359740813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.65200000000000002,"5min":0.751,"15min":0.79000000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.48499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72299999999999998,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.60099999999999998},"min":{"1min":0.47999999999999998,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.70699999999999996,"5min":0.71999999999999997,"15min":0.81599999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.745,"5min":0.76400000000000001,"15min":0.82999999999999996},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:07:16.793 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.793 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.793 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.793 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.794 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.794 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.794 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.794 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.794 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.794 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:16.795 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:07:17.734 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:17.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:17.764 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:17.760+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:18.707 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:18.704+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:18.729 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:18.724+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:19.744 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:19.740+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:19.757 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:19.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:20.698 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:20.692+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:20.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:20.756+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:21.717 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:21.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:21.779 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:21.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:22.747 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:22.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:22.781 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:22.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:23.743 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:23.736+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:23.783 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:23.780+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:24.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:24.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:24.770 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:24.764+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:25.734 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:25.728+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:25.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:25.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:26.744 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:26.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:26.775 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:26.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:27.743 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:27.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:27.782 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:27.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:28.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:28.696+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:28.775 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:28.768+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:29.713 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:29.708+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:29.786 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:29.780+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:30.725 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:30.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:30.819 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:30.812+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:31.750 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:31.744+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:31.829 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:31.824+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:32.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:32.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:32.845 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:32.840+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:33.762 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:33.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:33.802 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:33.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:34.726 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:34.720+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:34.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:34.752+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:35.721 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:35.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:35.789 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:35.784+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:36.732 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:36.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:36.778 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:36.772+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:36.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:07:37.089 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:07:37.091 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:07:37.101 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6261,"stamp":"2024-01-12T01:07:35.953923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003376"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1981,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.357802+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:04:28.357802+0000","last_peered":"2024-01-12T01:04:28.357802+0000","last_clean":"2024-01-12T01:04:28.357802+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:04:28.357802+0000","last_undegraded":"2024-01-12T01:04:28.357802+0000","last_fullsized":"2024-01-12T01:04:28.357802+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1714,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.650810+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:04:28.650810+0000","last_peered":"2024-01-12T01:04:28.650810+0000","last_clean":"2024-01-12T01:04:28.650810+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:04:28.650810+0000","last_undegraded":"2024-01-12T01:04:28.650810+0000","last_fullsized":"2024-01-12T01:04:28.650810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1926,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.942334+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:04:28.942334+0000","last_peered":"2024-01-12T01:04:28.942334+0000","last_clean":"2024-01-12T01:04:28.942334+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:04:28.942334+0000","last_undegraded":"2024-01-12T01:04:28.942334+0000","last_fullsized":"2024-01-12T01:04:28.942334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12872,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.235425+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:04:29.235425+0000","last_peered":"2024-01-12T01:04:29.235425+0000","last_clean":"2024-01-12T01:04:29.235425+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:04:29.235425+0000","last_undegraded":"2024-01-12T01:04:29.235425+0000","last_fullsized":"2024-01-12T01:04:29.235425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2145,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:04:29.529311+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:04:29.529311+0000","last_peered":"2024-01-12T01:04:29.529311+0000","last_clean":"2024-01-12T01:04:29.529311+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:04:29.529311+0000","last_undegraded":"2024-01-12T01:04:29.529311+0000","last_fullsized":"2024-01-12T01:04:29.529311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 765s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12356,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.818668+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:04:29.818668+0000","last_peered":"2024-01-12T01:04:29.818668+0000","last_clean":"2024-01-12T01:04:29.818668+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:04:29.818668+0000","last_undegraded":"2024-01-12T01:04:29.818668+0000","last_fullsized":"2024-01-12T01:04:29.818668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7507,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.127871+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:04:30.127871+0000","last_peered":"2024-01-12T01:04:30.127871+0000","last_clean":"2024-01-12T01:04:30.127871+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:04:30.127871+0000","last_undegraded":"2024-01-12T01:04:30.127871+0000","last_fullsized":"2024-01-12T01:04:30.127871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11784,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.423668+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:04:30.423668+0000","last_peered":"2024-01-12T01:04:30.423668+0000","last_clean":"2024-01-12T01:04:30.423668+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:04:30.423668+0000","last_undegraded":"2024-01-12T01:04:30.423668+0000","last_fullsized":"2024-01-12T01:04:30.423668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2522,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.722844+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:04:30.722844+0000","last_peered":"2024-01-12T01:04:30.722844+0000","last_clean":"2024-01-12T01:04:30.722844+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:04:30.722844+0000","last_undegraded":"2024-01-12T01:04:30.722844+0000","last_fullsized":"2024-01-12T01:04:30.722844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:31.013211+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:04:31.013211+0000","last_peered":"2024-01-12T01:04:31.013211+0000","last_clean":"2024-01-12T01:04:31.013211+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:04:31.013211+0000","last_undegraded":"2024-01-12T01:04:31.013211+0000","last_fullsized":"2024-01-12T01:04:31.013211+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740816,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54500000000000004,"15min":0.53600000000000003},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.32600000000000001},"max":{"1min":0.67000000000000004,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.629},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.45800000000000002,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.754,"15min":1.028},"last":0.58099999999999996}]},{"osd":1,"last update":"Fri Jan 12 01:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.58099999999999996},"min":{"1min":0.504,"5min":0.47099999999999997,"15min":0.374},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.86099999999999999,"15min":0.92700000000000005},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359740814,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52000000000000002,"15min":0.51500000000000001},"min":{"1min":0.43099999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.72699999999999998,"5min":0.81399999999999995,"15min":0.875},"last":0.56899999999999995}]},{"osd":2,"last update":"Fri Jan 12 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.626,"15min":0.59999999999999998},"min":{"1min":0.54600000000000004,"5min":0.46600000000000003,"15min":0.439},"max":{"1min":0.752,"5min":0.877,"15min":0.877},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.71199999999999997,"15min":0.68100000000000005},"min":{"1min":0.626,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.81100000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.64100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.65200000000000002,"5min":0.751,"15min":0.79000000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.48499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72299999999999998,"5min":0.82299999999999995,"15min":0.94799999999999995},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Jan 12 01:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.60099999999999998},"min":{"1min":0.47999999999999998,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.70699999999999996,"5min":0.71999999999999997,"15min":0.81599999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.745,"5min":0.76400000000000001,"15min":0.82999999999999996},"last":0.60699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:07:37.103 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.103 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.103 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.103 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.103 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.103 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.104 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.104 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.104 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.104 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:37.104 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:07:37.720 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:37.716+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:37.803 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:37.800+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:38.742 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:38.736+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:38.840 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:38.836+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:39.755 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:39.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:39.859 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:39.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:40.756 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:40.756+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:40.891 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:40.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:41.758 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:41.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:41.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:41.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:42.772 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:42.772+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:42.851 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:42.852+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:43.731 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:43.728+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:43.888 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:43.888+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:44.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:44.748+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:44.858 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:44.856+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:45.712 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:45.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:45.824 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:45.824+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:46.700 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:46.700+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:46.817 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:46.816+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:47.711 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:47.712+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:47.868 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:47.868+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:48.752 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:48.752+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:48.832 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:48.832+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:49.760 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:49.760+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:49.843 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:49.844+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:50.795 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:50.796+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:50.807 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:50.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:51.777 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:51.776+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:51.824 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:51.824+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:52.792 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:52.792+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:52.810 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:52.808+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:53.766 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:53.764+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:53.833 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:53.832+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:54.742 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:54.740+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:54.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:54.808+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:55.710 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:55.707+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:55.819 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:55.819+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:56.669 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:56.667+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:56.804 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:56.803+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:57.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:07:57.405 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:07:57.407 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:07:57.418 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6271,"stamp":"2024-01-12T01:07:55.959098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003127"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1981,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.357802+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:04:28.357802+0000","last_peered":"2024-01-12T01:04:28.357802+0000","last_clean":"2024-01-12T01:04:28.357802+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:04:28.357802+0000","last_undegraded":"2024-01-12T01:04:28.357802+0000","last_fullsized":"2024-01-12T01:04:28.357802+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1714,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.650810+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:04:28.650810+0000","last_peered":"2024-01-12T01:04:28.650810+0000","last_clean":"2024-01-12T01:04:28.650810+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:04:28.650810+0000","last_undegraded":"2024-01-12T01:04:28.650810+0000","last_fullsized":"2024-01-12T01:04:28.650810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1926,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.942334+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:04:28.942334+0000","last_peered":"2024-01-12T01:04:28.942334+0000","last_clean":"2024-01-12T01:04:28.942334+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:04:28.942334+0000","last_undegraded":"2024-01-12T01:04:28.942334+0000","last_fullsized":"2024-01-12T01:04:28.942334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12872,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.235425+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:04:29.235425+0000","last_peered":"2024-01-12T01:04:29.235425+0000","last_clean":"2024-01-12T01:04:29.235425+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:04:29.235425+0000","last_undegraded":"2024-01-12T01:04:29.235425+0000","last_fullsized":"2024-01-12T01:04:29.235425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2145,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:04:29.529311+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:04:29.529311+0000","last_peered":"2024-01-12T01:04:29.529311+0000","last_clean":"2024-01-12T01:04:29.529311+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:04:29.529311+0000","last_undegraded":"2024-01-12T01:04:29.529311+0000","last_fullsized":"2024-01-12T01:04:29.529311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 765s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12356,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.818668+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:04:29.818668+0000","last_peered":"2024-01-12T01:04:29.818668+0000","last_clean":"2024-01-12T01:04:29.818668+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:04:29.818668+0000","last_undegraded":"2024-01-12T01:04:29.818668+0000","last_fullsized":"2024-01-12T01:04:29.818668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7507,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.127871+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:04:30.127871+0000","last_peered":"2024-01-12T01:04:30.127871+0000","last_clean":"2024-01-12T01:04:30.127871+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:04:30.127871+0000","last_undegraded":"2024-01-12T01:04:30.127871+0000","last_fullsized":"2024-01-12T01:04:30.127871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11784,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.423668+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:04:30.423668+0000","last_peered":"2024-01-12T01:04:30.423668+0000","last_clean":"2024-01-12T01:04:30.423668+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:04:30.423668+0000","last_undegraded":"2024-01-12T01:04:30.423668+0000","last_fullsized":"2024-01-12T01:04:30.423668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2522,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.722844+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:04:30.722844+0000","last_peered":"2024-01-12T01:04:30.722844+0000","last_clean":"2024-01-12T01:04:30.722844+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:04:30.722844+0000","last_undegraded":"2024-01-12T01:04:30.722844+0000","last_fullsized":"2024-01-12T01:04:30.722844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:31.013211+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:04:31.013211+0000","last_peered":"2024-01-12T01:04:31.013211+0000","last_clean":"2024-01-12T01:04:31.013211+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:04:31.013211+0000","last_undegraded":"2024-01-12T01:04:31.013211+0000","last_fullsized":"2024-01-12T01:04:31.013211+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740820,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54500000000000004,"15min":0.53600000000000003},"min":{"1min":0.44700000000000001,"5min":0.442,"15min":0.32600000000000001},"max":{"1min":0.67000000000000004,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.45800000000000002,"5min":0.41099999999999998,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.754,"15min":1.028},"last":0.70299999999999996}]},{"osd":1,"last update":"Fri Jan 12 01:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.58099999999999996},"min":{"1min":0.504,"5min":0.47099999999999997,"15min":0.374},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.86099999999999999,"15min":0.92700000000000005},"last":0.59499999999999997}]}]},{"osd":1,"up_from":8,"seq":34359740818,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52000000000000002,"15min":0.51500000000000001},"min":{"1min":0.43099999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.5},{"interface":"front","average":{"1min":0.627,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.72699999999999998,"5min":0.81399999999999995,"15min":0.875},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.626,"15min":0.59999999999999998},"min":{"1min":0.54600000000000004,"5min":0.46600000000000003,"15min":0.439},"max":{"1min":0.752,"5min":0.877,"15min":0.877},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.71199999999999997,"15min":0.68100000000000005},"min":{"1min":0.626,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.81100000000000005,"5min":0.93500000000000005,"15min":1.071},"last":0.72499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359740821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.41299999999999998,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.79000000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.46899999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.94799999999999995},"last":0.57299999999999995}]},{"osd":2,"last update":"Fri Jan 12 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.81599999999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.499,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.82999999999999996},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:07:57.419 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.419 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.419 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.419 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.420 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.420 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.420 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.420 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.420 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.420 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:07:57.421 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-12T01:07:57.678 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:57.675+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:57.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:57.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:58.659 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:58.659+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:58.757 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:58.755+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:59.688 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:07:59.687+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:07:59.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:07:59.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:00.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:00.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:00.794 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:00.791+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:01.716 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:01.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:01.788 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:01.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:02.696 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:02.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:02.780 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:02.779+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:03.699 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:03.695+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:03.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:03.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:04.706 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:04.703+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:04.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:04.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:05.755 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:05.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:05.789 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:05.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:06.748 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:06.747+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:06.784 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:06.783+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:07.752 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:07.751+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:07.753 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:07.751+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:08.726 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:08.723+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:08.769 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:08.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:09.715 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:09.715+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:09.790 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:09.787+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:10.763 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:10.759+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:10.768 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:10.767+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:11.758 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:11.755+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:11.810 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:11.807+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:12.714 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:12.711+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:12.802 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:12.799+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:13.679 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:13.679+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:13.848 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:13.847+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:14.693 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:14.691+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:14.875 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:14.871+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:15.721 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:15.719+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:15.834 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:15.831+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:16.741 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:16.739+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:16.825 INFO:tasks.ceph.osd.0.smithi178.stderr:2024-01-12T01:08:16.823+0000 7f6c36f0e700 -1 osd.0 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:2647 3.3 3:e0fd5656:test1::rbd_data.2951ff622ce2.0000000000000076:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~4096 [fadvise_random] in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:17.422 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-12T01:08:17.716 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-12T01:08:17.718 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-01-12T01:08:17.729 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":6281,"stamp":"2024-01-12T01:08:15.964640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":726948487,"num_objects":464,"num_object_clones":2,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":167,"num_read":65833,"num_read_kb":141249,"num_write":19613,"num_write_kb":8859080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27531,"ondisk_log_size":27531,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294736,"kb_used_data":370104,"kb_used_omap":0,"kb_used_meta":924608,"kb_avail":281820784,"statfs":{"total":289910292480,"available":288584482816,"internally_reserved":0,"allocated":378986496,"data_stored":376522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":946798592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,512],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003241"},"pg_stats":[{"pgid":"2.7","version":"160'1074","reported_seq":1981,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.357802+0000","last_change":"2024-01-11T21:49:51.639192+0000","last_active":"2024-01-12T01:04:28.357802+0000","last_peered":"2024-01-12T01:04:28.357802+0000","last_clean":"2024-01-12T01:04:28.357802+0000","last_became_active":"2024-01-11T21:44:21.487435+0000","last_became_peered":"2024-01-11T21:44:21.487435+0000","last_unstale":"2024-01-12T01:04:28.357802+0000","last_undegraded":"2024-01-12T01:04:28.357802+0000","last_fullsized":"2024-01-12T01:04:28.357802+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014775649999999999,"stat_sum":{"num_bytes":102390484,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":2,"num_read":682,"num_read_kb":2032,"num_write":934,"num_write_kb":1119752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'854","reported_seq":1714,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.650810+0000","last_change":"2024-01-11T21:49:51.637936+0000","last_active":"2024-01-12T01:04:28.650810+0000","last_peered":"2024-01-12T01:04:28.650810+0000","last_clean":"2024-01-12T01:04:28.650810+0000","last_became_active":"2024-01-11T21:44:21.487700+0000","last_became_peered":"2024-01-11T21:44:21.487700+0000","last_unstale":"2024-01-12T01:04:28.650810+0000","last_undegraded":"2024-01-12T01:04:28.650810+0000","last_fullsized":"2024-01-12T01:04:28.650810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009915779999999999,"stat_sum":{"num_bytes":46198950,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":537,"num_read_kb":5427,"num_write":681,"num_write_kb":1059558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'1038","reported_seq":1926,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:28.942334+0000","last_change":"2024-01-11T21:49:51.638709+0000","last_active":"2024-01-12T01:04:28.942334+0000","last_peered":"2024-01-12T01:04:28.942334+0000","last_clean":"2024-01-12T01:04:28.942334+0000","last_became_active":"2024-01-11T21:44:21.489186+0000","last_became_peered":"2024-01-11T21:44:21.489186+0000","last_unstale":"2024-01-12T01:04:28.942334+0000","last_undegraded":"2024-01-12T01:04:28.942334+0000","last_fullsized":"2024-01-12T01:04:28.942334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358936,"stat_sum":{"num_bytes":101470258,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":626,"num_read_kb":1193,"num_write":880,"num_write_kb":1116515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'900","reported_seq":1841,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:53:45.293750+0000","last_change":"2024-01-12T00:53:45.293690+0000","last_active":"2024-01-12T00:53:45.293750+0000","last_peered":"2024-01-12T00:53:45.293750+0000","last_clean":"2024-01-12T00:53:45.293750+0000","last_became_active":"2024-01-11T21:44:21.489535+0000","last_became_peered":"2024-01-11T21:44:21.489535+0000","last_unstale":"2024-01-12T00:53:45.293750+0000","last_undegraded":"2024-01-12T00:53:45.293750+0000","last_fullsized":"2024-01-12T00:53:45.293750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'900","last_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_deep_scrub":"163'900","last_deep_scrub_stamp":"2024-01-12T00:53:45.293478+0000","last_clean_scrub_stamp":"2024-01-12T00:53:45.293478+0000","objects_scrubbed":7,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:43:46.308115+0000","scrub_duration":0.030486426,"objects_trimmed":0,"snaptrim_duration":0.001223933,"stat_sum":{"num_bytes":4194461,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":591,"num_read_kb":1345,"num_write":745,"num_write_kb":1117571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":148,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'5427","reported_seq":12872,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.235425+0000","last_change":"2024-01-11T21:49:51.648101+0000","last_active":"2024-01-12T01:04:29.235425+0000","last_peered":"2024-01-12T01:04:29.235425+0000","last_clean":"2024-01-12T01:04:29.235425+0000","last_became_active":"2024-01-11T21:44:27.572118+0000","last_became_peered":"2024-01-11T21:44:27.572118+0000","last_unstale":"2024-01-12T01:04:29.235425+0000","last_undegraded":"2024-01-12T01:04:29.235425+0000","last_fullsized":"2024-01-12T01:04:29.235425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5427,"log_dups_size":0,"ondisk_log_size":5427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081469799999999998,"stat_sum":{"num_bytes":40327388,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":43,"num_read":19902,"num_read_kb":35397,"num_write":4979,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2268,"num_flush":201,"num_flush_kb":4866,"num_evict":557,"num_evict_kb":37869,"num_promote":630,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'1037","reported_seq":2145,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-12T01:04:29.529311+0000","last_change":"2024-01-12T00:53:48.814046+0000","last_active":"2024-01-12T01:04:29.529311+0000","last_peered":"2024-01-12T01:04:29.529311+0000","last_clean":"2024-01-12T01:04:29.529311+0000","last_became_active":"2024-01-11T21:44:21.488440+0000","last_became_peered":"2024-01-11T21:44:21.488440+0000","last_unstale":"2024-01-12T01:04:29.529311+0000","last_undegraded":"2024-01-12T01:04:29.529311+0000","last_fullsized":"2024-01-12T01:04:29.529311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 765s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059575000000000003,"stat_sum":{"num_bytes":36409533,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":729,"num_read_kb":1262,"num_write":885,"num_write_kb":1227440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'5558","reported_seq":12356,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:29.818668+0000","last_change":"2024-01-11T21:49:51.642688+0000","last_active":"2024-01-12T01:04:29.818668+0000","last_peered":"2024-01-12T01:04:29.818668+0000","last_clean":"2024-01-12T01:04:29.818668+0000","last_became_active":"2024-01-11T21:44:27.572042+0000","last_became_peered":"2024-01-11T21:44:27.572042+0000","last_unstale":"2024-01-12T01:04:29.818668+0000","last_undegraded":"2024-01-12T01:04:29.818668+0000","last_fullsized":"2024-01-12T01:04:29.818668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049391490000000003,"stat_sum":{"num_bytes":8152558,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":45,"num_read":16313,"num_read_kb":44799,"num_write":3615,"num_write_kb":1883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":1420,"num_flush":214,"num_flush_kb":1118,"num_evict":637,"num_evict_kb":56239,"num_promote":685,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"163'1116","reported_seq":2073,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:24.947814+0000","last_change":"2024-01-12T00:51:24.947773+0000","last_active":"2024-01-12T00:51:24.947814+0000","last_peered":"2024-01-12T00:51:24.947814+0000","last_clean":"2024-01-12T00:51:24.947814+0000","last_became_active":"2024-01-11T21:44:21.489821+0000","last_became_peered":"2024-01-11T21:44:21.489821+0000","last_unstale":"2024-01-12T00:51:24.947814+0000","last_undegraded":"2024-01-12T00:51:24.947814+0000","last_fullsized":"2024-01-12T00:51:24.947814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1116","last_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_deep_scrub":"163'1116","last_deep_scrub_stamp":"2024-01-12T00:51:24.947629+0000","last_clean_scrub_stamp":"2024-01-12T00:51:24.947629+0000","objects_scrubbed":40,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:54:41.878930+0000","scrub_duration":0.17186196000000001,"objects_trimmed":0,"snaptrim_duration":0.0011520199999999999,"stat_sum":{"num_bytes":134025286,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":790,"num_read_kb":1540,"num_write":929,"num_write_kb":1070553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'3000","reported_seq":7507,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.127871+0000","last_change":"2024-01-11T21:49:51.651383+0000","last_active":"2024-01-12T01:04:30.127871+0000","last_peered":"2024-01-12T01:04:30.127871+0000","last_clean":"2024-01-12T01:04:30.127871+0000","last_became_active":"2024-01-11T21:44:27.573613+0000","last_became_peered":"2024-01-11T21:44:27.573613+0000","last_unstale":"2024-01-12T01:04:30.127871+0000","last_undegraded":"2024-01-12T01:04:30.127871+0000","last_fullsized":"2024-01-12T01:04:30.127871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033789829999999999,"stat_sum":{"num_bytes":38793880,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":38,"num_read":9720,"num_read_kb":14327,"num_write":1653,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":664,"num_flush":160,"num_flush_kb":15868,"num_evict":540,"num_evict_kb":25930,"num_promote":602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"163'1440","reported_seq":2384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T00:51:20.793156+0000","last_change":"2024-01-12T00:51:20.793124+0000","last_active":"2024-01-12T00:51:20.793156+0000","last_peered":"2024-01-12T00:51:20.793156+0000","last_clean":"2024-01-12T00:51:20.793156+0000","last_became_active":"2024-01-11T21:44:21.490683+0000","last_became_peered":"2024-01-11T21:44:21.490683+0000","last_unstale":"2024-01-12T00:51:20.793156+0000","last_undegraded":"2024-01-12T00:51:20.793156+0000","last_fullsized":"2024-01-12T00:51:20.793156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'1440","last_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_deep_scrub":"163'1440","last_deep_scrub_stamp":"2024-01-12T00:51:20.792997+0000","last_clean_scrub_stamp":"2024-01-12T00:51:20.792997+0000","objects_scrubbed":33,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-13T05:25:24.355574+0000","scrub_duration":0.095278304999999994,"objects_trimmed":0,"snaptrim_duration":0.001072927,"stat_sum":{"num_bytes":64401744,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":586,"num_read_kb":4599,"num_write":841,"num_write_kb":1034097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4412","reported_seq":11784,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.423668+0000","last_change":"2024-01-11T21:49:51.638810+0000","last_active":"2024-01-12T01:04:30.423668+0000","last_peered":"2024-01-12T01:04:30.423668+0000","last_clean":"2024-01-12T01:04:30.423668+0000","last_became_active":"2024-01-11T21:44:27.574464+0000","last_became_peered":"2024-01-11T21:44:27.574464+0000","last_unstale":"2024-01-12T01:04:30.423668+0000","last_undegraded":"2024-01-12T01:04:30.423668+0000","last_fullsized":"2024-01-12T01:04:30.423668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:26.492271+0000","last_clean_scrub_stamp":"2024-01-11T21:44:26.492271+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001265621,"stat_sum":{"num_bytes":33847300,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":39,"num_read":14667,"num_read_kb":26284,"num_write":2350,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":2033,"num_flush":200,"num_flush_kb":19793,"num_evict":664,"num_evict_kb":32058,"num_promote":716,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'1578","reported_seq":2522,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:30.722844+0000","last_change":"2024-01-11T21:49:51.638392+0000","last_active":"2024-01-12T01:04:30.722844+0000","last_peered":"2024-01-12T01:04:30.722844+0000","last_clean":"2024-01-12T01:04:30.722844+0000","last_became_active":"2024-01-11T21:44:21.487673+0000","last_became_peered":"2024-01-11T21:44:21.487673+0000","last_unstale":"2024-01-12T01:04:30.722844+0000","last_undegraded":"2024-01-12T01:04:30.722844+0000","last_fullsized":"2024-01-12T01:04:30.722844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:20.466901+0000","last_clean_scrub_stamp":"2024-01-11T21:44:20.466901+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393619,"stat_sum":{"num_bytes":116146277,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":584,"num_read_kb":2953,"num_write":984,"num_write_kb":1103433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-01-12T01:04:31.013211+0000","last_change":"2024-01-11T21:44:16.482782+0000","last_active":"2024-01-12T01:04:31.013211+0000","last_peered":"2024-01-12T01:04:31.013211+0000","last_clean":"2024-01-12T01:04:31.013211+0000","last_became_active":"2024-01-11T21:44:16.480508+0000","last_became_peered":"2024-01-11T21:44:16.480508+0000","last_unstale":"2024-01-12T01:04:31.013211+0000","last_undegraded":"2024-01-12T01:04:31.013211+0000","last_fullsized":"2024-01-12T01:04:31.013211+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T21:44:15.465734+0000","last_clean_scrub_stamp":"2024-01-11T21:44:15.465734+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":121121126,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":165,"num_read":60602,"num_read_kb":120807,"num_write":12597,"num_write_kb":8267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":6385,"num_flush":775,"num_flush_kb":41645,"num_evict":2398,"num_evict_kb":152096,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":3,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150577152,"data_stored":150344396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18397,"ondisk_log_size":18397,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":605236993,"num_objects":234,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":2,"num_read":5125,"num_read_kb":20351,"num_write":6879,"num_write_kb":8848919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":740,"num_omap_keys":81,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":222895618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9037,"ondisk_log_size":9037,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740824,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433092,"kb_used_data":123516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93938748,"statfs":{"total":96636764160,"available":96193277952,"internally_reserved":0,"allocated":126480384,"data_stored":125648235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.32600000000000001},"max":{"1min":0.629,"5min":0.76400000000000001,"15min":0.93899999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59099999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.70299999999999996,"5min":0.71899999999999997,"15min":1.028},"last":0.53600000000000003}]},{"osd":1,"last update":"Fri Jan 12 01:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.374},"max":{"1min":0.68100000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65000000000000002,"15min":0.67100000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.42299999999999999},"max":{"1min":0.746,"5min":0.86099999999999999,"15min":0.92700000000000005},"last":0.66000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359740822,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446616,"kb_used_data":137040,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93925224,"statfs":{"total":96636764160,"available":96179429376,"internally_reserved":0,"allocated":140328960,"data_stored":139485469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 12 01:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52000000000000002,"15min":0.51500000000000001},"min":{"1min":0.43099999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.39700000000000002},"max":{"1min":0.72699999999999998,"5min":0.81399999999999995,"15min":0.875},"last":0.67200000000000004}]},{"osd":2,"last update":"Fri Jan 12 01:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.439},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.877},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70699999999999996,"15min":0.68300000000000005},"min":{"1min":0.52700000000000002,"5min":0.39300000000000002,"15min":0.28299999999999997},"max":{"1min":0.83899999999999997,"5min":0.86899999999999999,"15min":1.071},"last":0.73999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415028,"kb_used_data":109548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93956812,"statfs":{"total":96636764160,"available":96211775488,"internally_reserved":0,"allocated":112177152,"data_stored":111389099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 12 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.41299999999999998,"5min":0.40500000000000003,"15min":0.38500000000000001},"max":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.79000000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.46899999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.94799999999999995},"last":0.69099999999999995}]},{"osd":2,"last update":"Fri Jan 12 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.81599999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.499,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.82999999999999996},"last":0.78200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61739008,"data_stored":61672710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111566848,"data_stored":111447809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49827840,"data_stored":49775099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48484352,"data_stored":48425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26804224,"data_stored":26746354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75288576,"data_stored":75172198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-12T01:08:17.730 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.730 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.731 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.731 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.731 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.731 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.731 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.732 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-01-11T21:44:26.492271+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=26, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.732 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-11T21:44:20.466901+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.732 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T21:44:15.465734+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=21, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=47, tm_sec=19, tm_wday=4, tm_yday=12, tm_isdst=0) 2024-01-12T01:08:17.732 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2024-01-12T01:08:17.743 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-12T01:08:17.743 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-12T01:08:17.743 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-12T01:08:17.743 INFO:teuthology.orchestra.run:waiting for 300 2024-01-12T01:08:17.762 INFO:tasks.ceph.osd.1.smithi178.stderr:2024-01-12T01:08:17.759+0000 7fdf89167700 -1 osd.1 163 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10580.0:527 3.0 3:2fa92bdb:test1::rbd_data.2951ff622ce2.000000000000006b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e160) 2024-01-12T01:08:17.808 INFO:tasks.ceph.osd.0:Stopped 2024-01-12T01:08:17.808 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-12T01:08:17.809 INFO:teuthology.orchestra.run:waiting for 300 2024-01-12T01:08:17.875 INFO:tasks.ceph.osd.1:Stopped 2024-01-12T01:08:17.875 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-12T01:08:17.876 INFO:teuthology.orchestra.run:waiting for 300 2024-01-12T01:08:17.951 INFO:tasks.ceph.osd.2:Stopped 2024-01-12T01:08:17.952 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-12T01:08:17.952 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-12T01:08:17.952 INFO:teuthology.orchestra.run:waiting for 300 2024-01-12T01:08:17.980 INFO:tasks.ceph.mgr.x:Stopped 2024-01-12T01:08:17.981 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-12T01:08:17.981 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-12T01:08:17.981 INFO:teuthology.orchestra.run:waiting for 300 2024-01-12T01:08:18.006 INFO:tasks.ceph.mon.a:Stopped 2024-01-12T01:08:18.006 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-12T01:08:18.006 DEBUG:teuthology.orchestra.run.smithi178:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 2024-01-12T01:08:18.025 INFO:teuthology.orchestra.run.smithi178.stdout:1705009841.7733831 osd.1 (osd.1) 3 : cluster [WRN] 6 slow requests (by type [ 'reached pg' : 5 'delayed' : 1 ] most affected pool [ 'cache' : 6 ]) 2024-01-12T01:08:18.027 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-01-12T01:08:18.027 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi178.front.sepia.ceph.com 2024-01-12T01:08:18.027 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-12T01:08:18.305 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi178.front.sepia.ceph.com 2024-01-12T01:08:18.305 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-12T01:08:18.587 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi178.front.sepia.ceph.com 2024-01-12T01:08:18.587 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-12T01:08:18.768 INFO:tasks.ceph:Archiving mon data... 2024-01-12T01:08:18.770 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820/data/mon.a.tgz 2024-01-12T01:08:18.770 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-01-12T01:08:18.776 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.0TPQNywEU5 2024-01-12T01:08:18.777 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0TPQNywEU5 2024-01-12T01:08:18.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.0TPQNywEU5 2024-01-12T01:08:18.938 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.0TPQNywEU5 is 419KB 2024-01-12T01:08:19.005 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.0TPQNywEU5 2024-01-12T01:08:19.012 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-12T01:08:19.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-12T01:08:19.028 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-01-12T01:08:19.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-12T01:08:19.060 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-12T01:08:19.066 INFO:tasks.ceph:Archiving crash dumps... 2024-01-12T01:08:19.068 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820/remote/smithi178/crash 2024-01-12T01:08:19.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-12T01:08:19.123 INFO:tasks.ceph:Compressing logs... 2024-01-12T01:08:19.123 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-12T01:08:19.138 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67016.log 2024-01-12T01:08:19.138 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24831.log 2024-01-12T01:08:19.139 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67016.log.gz 2024-01-12T01:08:19.139 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52451.log 2024-01-12T01:08:19.139 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24831.log: /var/log/ceph/ceph-client.admin.27808.log 2024-01-12T01:08:19.141 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24831.log.gz 2024-01-12T01:08:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68918.log 2024-01-12T01:08:19.141 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52451.log.gz 2024-01-12T01:08:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67145.log 2024-01-12T01:08:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27808.log: /var/log/ceph/ceph-client.admin.68918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27808.log.gz 2024-01-12T01:08:19.141 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68918.log.gz 2024-01-12T01:08:19.142 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69216.log 2024-01-12T01:08:19.142 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67145.log.gz 2024-01-12T01:08:19.143 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65352.log 2024-01-12T01:08:19.143 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69216.log.gz 2024-01-12T01:08:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55456.log 2024-01-12T01:08:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65352.log.gz 2024-01-12T01:08:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46103.log 2024-01-12T01:08:19.145 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30402.log 2024-01-12T01:08:19.145 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55456.log: /var/log/ceph/ceph-client.admin.46103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55456.log.gz 2024-01-12T01:08:19.145 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46103.log.gz 2024-01-12T01:08:19.146 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31388.log 2024-01-12T01:08:19.146 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46519.log 2024-01-12T01:08:19.146 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30402.log.gz 2024-01-12T01:08:19.147 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52236.log 2024-01-12T01:08:19.147 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31388.log: /var/log/ceph/ceph-client.admin.46519.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31388.log.gz 2024-01-12T01:08:19.147 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.46519.log.gz 2024-01-12T01:08:19.148 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65518.log 2024-01-12T01:08:19.148 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55219.log 2024-01-12T01:08:19.149 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52236.log.gz 2024-01-12T01:08:19.149 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65539.log 2024-01-12T01:08:19.149 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65518.log: /var/log/ceph/ceph-client.admin.55219.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.55219.log.gz 2024-01-12T01:08:19.149 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.51590.log 2024-01-12T01:08:19.149 INFO:teuthology.orchestra.run.smithi178.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.65518.log.gz 2024-01-12T01:08:19.150 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65539.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.65539.log.gz 2024-01-12T01:08:19.150 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33368.log 2024-01-12T01:08:19.150 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67339.log 2024-01-12T01:08:19.151 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59250.log 2024-01-12T01:08:19.151 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51590.log.gz 2024-01-12T01:08:19.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67339.log: /var/log/ceph/ceph-client.admin.33368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67339.log.gz 2024-01-12T01:08:19.152 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33368.log.gz 2024-01-12T01:08:19.152 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59313.log 2024-01-12T01:08:19.152 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70785.log /var/log/ceph/ceph-client.admin.59250.log: 2024-01-12T01:08:19.153 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59250.log.gz 2024-01-12T01:08:19.153 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59313.log.gz 2024-01-12T01:08:19.153 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37786.log 2024-01-12T01:08:19.154 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58286.log 2024-01-12T01:08:19.154 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70785.log.gz 2024-01-12T01:08:19.155 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57250.log 2024-01-12T01:08:19.155 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37786.log: /var/log/ceph/ceph-client.admin.58286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37786.log.gz 2024-01-12T01:08:19.155 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58286.log.gz 2024-01-12T01:08:19.155 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34894.log 2024-01-12T01:08:19.155 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57250.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.57250.log.gz 2024-01-12T01:08:19.155 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.49459.log 2024-01-12T01:08:19.157 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27186.log 2024-01-12T01:08:19.157 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34894.log: 0.0%/var/log/ceph/ceph-client.admin.49459.log: -- replaced with /var/log/ceph/ceph-client.admin.34894.log.gz 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.49459.log.gz 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33608.log 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.27186.log: --verbose -- /var/log/ceph/ceph-client.admin.51359.log 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27186.log.gz 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33608.log.gz 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61113.log 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41430.log 2024-01-12T01:08:19.158 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51359.log.gz 2024-01-12T01:08:19.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61113.log.gz 2024-01-12T01:08:19.159 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50598.log 2024-01-12T01:08:19.165 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52322.log 2024-01-12T01:08:19.165 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41430.log.gz 2024-01-12T01:08:19.165 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23756.log 2024-01-12T01:08:19.165 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50598.log: /var/log/ceph/ceph-client.admin.52322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50598.log.gz 2024-01-12T01:08:19.165 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47709.log 2024-01-12T01:08:19.165 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23756.log.gz 2024-01-12T01:08:19.165 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47967.log 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52322.log.gz 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31303.log 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47709.log.gz 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26285.log 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47967.log.gz 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47192.log 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31303.log.gz 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27572.log 2024-01-12T01:08:19.166 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26285.log.gz 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59410.log 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47192.log.gz 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.27572.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60875.log 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27572.log.gz 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43798.log 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59410.log.gz 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36374.log 2024-01-12T01:08:19.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60875.log.gz 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43798.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43798.log.gz 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.39337.log 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28367.log 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36374.log: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36374.log.gz 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39883.log 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39337.log.gz 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38620.log 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28367.log: 0.0%/var/log/ceph/ceph-client.admin.39883.log: -- replaced with /var/log/ceph/ceph-client.admin.28367.log.gz 2024-01-12T01:08:19.168 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39883.log.gz 2024-01-12T01:08:19.169 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62883.log 2024-01-12T01:08:19.182 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-01-12T01:08:19.182 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53917.log 2024-01-12T01:08:19.182 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33068.log 2024-01-12T01:08:19.182 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62883.log: /var/log/ceph/ceph-client.admin.53917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62883.log.gz 2024-01-12T01:08:19.182 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53917.log.gz 2024-01-12T01:08:19.182 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28453.log 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47924.log 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33068.log: /var/log/ceph/ceph-client.admin.28453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33068.log.gz 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36921.log 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28453.log.gz 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27250.log 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47924.log.gz 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70488.log 2024-01-12T01:08:19.183 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36921.log: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36921.log.gz 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35149.log 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27250.log.gz 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66394.log 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70488.log.gz 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35149.log.gz 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58461.log 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61052.log 2024-01-12T01:08:19.184 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66394.log.gz 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45548.log /var/log/ceph/ceph-client.admin.58461.log: 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58461.log.gz 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61052.log.gz 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36795.log 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56444.log 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45548.log.gz 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37362.log 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36795.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36795.log.gz 2024-01-12T01:08:19.185 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66415.log 2024-01-12T01:08:19.186 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56444.log.gz 2024-01-12T01:08:19.186 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37362.log: gzip 23.3%-5 -- replaced with /var/log/ceph/ceph-client.admin.37362.log.gz 2024-01-12T01:08:19.186 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.40006.log 2024-01-12T01:08:19.187 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43193.log 2024-01-12T01:08:19.187 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66415.log.gz 2024-01-12T01:08:19.187 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40006.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40006.log.gz-5 2024-01-12T01:08:19.187 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.39638.log 2024-01-12T01:08:19.187 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26564.log 2024-01-12T01:08:19.187 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43193.log.gz 2024-01-12T01:08:19.187 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51399.log 2024-01-12T01:08:19.188 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.39638.log.gz 2024-01-12T01:08:19.188 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57529.log 2024-01-12T01:08:19.188 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26564.log: /var/log/ceph/ceph-client.admin.51399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51399.log.gz 2024-01-12T01:08:19.188 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.26564.log.gz-5 2024-01-12T01:08:19.189 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.53853.log 2024-01-12T01:08:19.189 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47666.log 2024-01-12T01:08:19.189 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57529.log.gz 2024-01-12T01:08:19.189 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31848.log 2024-01-12T01:08:19.189 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53853.log: /var/log/ceph/ceph-client.admin.47666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53853.log.gz 2024-01-12T01:08:19.189 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35681.log 2024-01-12T01:08:19.189 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47666.log.gz 2024-01-12T01:08:19.190 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37915.log 2024-01-12T01:08:19.190 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31848.log.gz 2024-01-12T01:08:19.190 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58347.log 2024-01-12T01:08:19.190 INFO:teuthology.orchestra.run.smithi178.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.35681.log.gz 2024-01-12T01:08:19.190 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34231.log 2024-01-12T01:08:19.190 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37915.log.gz 2024-01-12T01:08:19.190 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23919.log 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58347.log.gz 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56464.log 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34231.log.gz 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23919.log.gz 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47256.log 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56464.log.gz 2024-01-12T01:08:19.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47256.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47256.log.gz 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.64099.log 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35064.log 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66887.log 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64099.log.gz 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47558.log 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35064.log.gz 2024-01-12T01:08:19.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66243.log 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66887.log.gz 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47558.log.gz 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43365.log 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66930.log 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66243.log.gz 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.43365.log: --verbose -- /var/log/ceph/ceph-client.admin.30640.log 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43365.log.gz 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66930.log.gz 2024-01-12T01:08:19.193 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69162.log 2024-01-12T01:08:19.730 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60040.log 2024-01-12T01:08:19.730 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30640.log.gz 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37047.log 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69162.log.gz 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65087.log 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60040.log.gz 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32288.log 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37047.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.37047.log.gz 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64873.log 2024-01-12T01:08:19.731 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65087.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.33388.log -- replaced with /var/log/ceph/ceph-client.admin.65087.log.gz 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32288.log.gz 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64873.log.gz 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27465.log 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33388.log.gz 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34384.log 2024-01-12T01:08:19.732 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42071.log 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27465.log.gz 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56260.log 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34384.log.gz 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42071.log.gz 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44457.log 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37509.log 2024-01-12T01:08:19.733 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56260.log.gz 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57895.log 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44457.log.gz 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56239.log 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37509.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.37509.log.gz 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69324.log 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57895.log.gz 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56239.log: /var/log/ceph/ceph-client.admin.40124.log 2024-01-12T01:08:19.734 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56239.log.gz 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69324.log: /var/log/ceph/ceph-client.admin.61618.log 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69324.log.gz 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28711.log 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40124.log.gz 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44164.log 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61618.log.gz 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38130.log 2024-01-12T01:08:19.735 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28711.log.gz 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42784.log 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44164.log.gz 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53509.log 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38130.log.gz 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65066.log 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42784.log.gz 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49858.log 2024-01-12T01:08:19.736 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53509.log.gz 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26263.log 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65066.log.gz 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60586.log 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49858.log.gz 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.26263.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32348.log 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26263.log.gz 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37551.log 2024-01-12T01:08:19.737 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60586.log.gz 2024-01-12T01:08:19.738 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-12T01:08:19.738 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32348.log.gz 2024-01-12T01:08:19.738 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34537.log 2024-01-12T01:08:19.738 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37551.log.gz 2024-01-12T01:08:19.738 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58088.log 2024-01-12T01:08:19.738 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2024-01-12T01:08:19.738 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34537.log.gz 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61404.log 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58088.log.gz 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64658.log 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55611.log 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61404.log: 53.7% -- replaced with /var/log/ceph/ceph-client.admin.61404.log.gz 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60740.log 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64658.log.gz 2024-01-12T01:08:19.739 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49880.log 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55611.log.gz 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.60740.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49200.log 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60740.log.gz 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47945.log 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49880.log.gz 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45715.log 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49200.log.gz 2024-01-12T01:08:19.740 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68999.log 2024-01-12T01:08:19.741 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47945.log.gz 2024-01-12T01:08:19.741 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55477.log 2024-01-12T01:08:19.741 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45715.log.gz 2024-01-12T01:08:19.741 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59179.log 2024-01-12T01:08:19.741 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68999.log.gz 2024-01-12T01:08:19.741 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65204.log 2024-01-12T01:08:19.741 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55477.log.gz 2024-01-12T01:08:19.742 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52666.log 2024-01-12T01:08:19.742 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59179.log.gz 2024-01-12T01:08:19.742 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51676.log 2024-01-12T01:08:19.742 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65204.log.gz 2024-01-12T01:08:19.742 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61309.log 2024-01-12T01:08:19.742 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52666.log.gz 2024-01-12T01:08:19.742 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.51676.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49923.log 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51676.log.gz 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68456.log 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61309.log.gz 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43648.log 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49923.log.gz 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36648.log 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68456.log.gz 2024-01-12T01:08:19.743 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61501.log 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43648.log.gz 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61287.log 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36648.log: /var/log/ceph/ceph-client.admin.61501.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.36648.log.gz 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39186.log 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61501.log.gz 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53982.log 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.61287.log.gz 2024-01-12T01:08:19.744 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29037.log 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39186.log.gz 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25263.log 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53982.log.gz 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66050.log 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29037.log.gz 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.25263.log: --verbose -- /var/log/ceph/ceph-client.admin.45226.log 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25263.log.gz 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53810.log 2024-01-12T01:08:19.745 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66050.log.gz 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61481.log 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45226.log.gz 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61152.log 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53810.log.gz 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61481.log.gz 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44246.log 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.61152.log.gz 2024-01-12T01:08:19.746 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56344.log 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65717.log 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44246.log.gz 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25535.log 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56344.log.gz 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57443.log 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65717.log.gz 2024-01-12T01:08:19.747 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59213.log 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25535.log.gz 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43863.log 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57443.log.gz 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59213.log.gz 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58538.log 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57744.log 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63798.log 2024-01-12T01:08:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58538.log: /var/log/ceph/ceph-client.admin.43863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58538.log.gz 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43863.log.gz 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57744.log.gz 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50478.log 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67970.log 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63798.log.gz 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34163.log 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50478.log.gz 2024-01-12T01:08:19.749 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55751.log 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67970.log.gz 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67274.log 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34163.log.gz 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55751.log.gz 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35474.log 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67274.log.gz 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66523.log 2024-01-12T01:08:19.750 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38799.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.35474.log: -- /var/log/ceph/ceph-client.admin.46373.log 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35474.log.gz 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44263.log 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67425.log 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66523.log.gz 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46373.log.gz 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39780.log 2024-01-12T01:08:19.751 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44263.log.gz 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67425.log.gz 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39780.log.gz 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36585.log 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68348.log 2024-01-12T01:08:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67666.log 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36585.log.gz 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.68348.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44775.log 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68348.log.gz 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35891.log 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67666.log: 11.8% -- replaced with /var/log/ceph/ceph-client.admin.67666.log.gz 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62474.log 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44775.log.gz 2024-01-12T01:08:19.753 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62152.log 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35891.log: 23.6% -- replaced with /var/log/ceph/ceph-client.admin.35891.log.gz 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37299.log 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62474.log.gz 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53320.log 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62152.log.gz 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29019.log 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37299.log.gz 2024-01-12T01:08:19.754 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50116.log 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53320.log.gz 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29019.log.gz 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51955.log 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66544.log 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50116.log.gz 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34758.log 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51955.log.gz 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51526.log 2024-01-12T01:08:19.755 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66544.log.gz 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64808.log 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34758.log.gz 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38388.log 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51526.log.gz 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64808.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64808.log.gz-5 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.25089.log 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25875.log 2024-01-12T01:08:19.756 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38388.log.gz 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53112.log 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25089.log.gz 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29336.log 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25875.log.gz 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56161.log 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53112.log.gz 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31728.log 2024-01-12T01:08:19.757 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29336.log.gz 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35702.log 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56161.log.gz 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31728.log: 2.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.31728.log.gz-5 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.48813.log 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35702.log: /var/log/ceph/ceph-client.admin.34520.log 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.35702.log.gz 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49966.log 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48813.log.gz 2024-01-12T01:08:19.758 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48535.log 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34520.log.gz 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45140.log 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49966.log.gz 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70950.log 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48535.log.gz 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62689.log 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45140.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45140.log.gz-5 2024-01-12T01:08:19.759 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.36059.log 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70950.log.gz 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44968.log 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62689.log.gz 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35597.log 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36059.log: 21.2% -- replaced with /var/log/ceph/ceph-client.admin.36059.log.gz 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35438.log 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44968.log.gz 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2024-01-12T01:08:19.760 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.35597.log.gz 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61444.log 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35438.log.gz 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50782.log 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32328.log 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61444.log.gz 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29098.log 2024-01-12T01:08:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50782.log.gz 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49028.log 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32328.log.gz 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28195.log 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29098.log: /var/log/ceph/ceph-client.admin.49028.log: gzip -5 --verbose -- 23.4%/var/log/ceph/ceph-client.admin.38637.log 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29098.log.gz 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49028.log.gz 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.28195.log: -- /var/log/ceph/ceph-client.admin.29119.log 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28195.log.gz 2024-01-12T01:08:19.762 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43085.log 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38637.log.gz 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50267.log 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.29119.log.gz 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37131.log 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43085.log.gz 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53009.log 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50267.log.gz 2024-01-12T01:08:19.763 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25649.log 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37131.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.37131.log.gz 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55520.log 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53009.log.gz 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40917.log 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25649.log.gz 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24573.log 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55520.log.gz 2024-01-12T01:08:19.764 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27722.log 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40917.log.gz 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61578.log 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24573.log.gz 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71356.log 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27722.log.gz 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35030.log 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61578.log.gz 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43236.log 2024-01-12T01:08:19.765 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71356.log.gz 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28775.log 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35030.log.gz 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68429.log 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43236.log.gz 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59196.log 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28775.log.gz 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50762.log 2024-01-12T01:08:19.766 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68429.log.gz 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58556.log 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59196.log.gz 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33748.log 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50762.log.gz 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26908.log 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58556.log.gz 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45897.log 2024-01-12T01:08:19.767 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33748.log.gz 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37614.log 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26908.log.gz 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26865.log 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45897.log.gz 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.37614.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.37614.log.gz 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67102.log 2024-01-12T01:08:19.768 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26865.log: /var/log/ceph/ceph-client.admin.39552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26865.log.gz 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44839.log 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30555.log 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67102.log.gz 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63669.log 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44839.log.gz 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26650.log 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30555.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30555.log.gz 2024-01-12T01:08:19.769 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24852.log 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63669.log.gz 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38993.log 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26650.log.gz 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.24852.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40523.log 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24852.log.gz 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33823.log 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38993.log.gz 2024-01-12T01:08:19.770 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43214.log 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40523.log.gz 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70298.log 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33823.log.gz 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64679.log 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43214.log.gz 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70298.log.gz 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66781.log 2024-01-12T01:08:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64679.log.gz 2024-01-12T01:08:19.772 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26242.log 2024-01-12T01:08:19.772 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-01-12T01:08:19.772 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61385.log 2024-01-12T01:08:19.772 INFO:teuthology.orchestra.run.smithi178.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.66781.log.gz 2024-01-12T01:08:19.772 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46576.log 2024-01-12T01:08:19.772 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26242.log.gz 2024-01-12T01:08:19.772 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51719.log 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61385.log.gz 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45355.log 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46576.log.gz 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63084.log 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51719.log.gz 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28947.log 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45355.log.gz 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59584.log 2024-01-12T01:08:19.773 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63084.log.gz 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34367.log 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28947.log.gz 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59584.log: /var/log/ceph/ceph-client.admin.51869.log 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59584.log.gz 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.34367.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42328.log 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34367.log.gz 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52988.log 2024-01-12T01:08:19.774 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51869.log.gz 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.42328.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25477.log 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42328.log.gz 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40875.log 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52988.log.gz 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37152.log 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25477.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25477.log.gz 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.40875.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33248.log 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2024-01-12T01:08:19.775 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2024-01-12T01:08:19.776 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37152.log: gzip -5 23.0%--verbose -- /var/log/ceph/ceph-client.admin.52279.log -- replaced with /var/log/ceph/ceph-client.admin.37152.log.gz 2024-01-12T01:08:19.776 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:19.776 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33248.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33248.log.gz-5 2024-01-12T01:08:19.776 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.56103.log 2024-01-12T01:08:19.776 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2024-01-12T01:08:19.776 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63991.log 2024-01-12T01:08:19.776 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52279.log.gz 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65124.log 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56103.log.gz 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51848.log 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63991.log.gz 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49665.log 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr: 12.9% -- replaced with /var/log/ceph/ceph-client.admin.65124.log.gz 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49772.log 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51848.log.gz 2024-01-12T01:08:19.777 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49007.log 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49665.log.gz 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.49772.log: -- /var/log/ceph/ceph-client.admin.58848.log 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49772.log.gz 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58067.log 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49007.log.gz 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38907.log 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58848.log.gz 2024-01-12T01:08:19.778 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.58067.log: --verbose -- /var/log/ceph/ceph-client.admin.48139.log 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58067.log.gz 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51740.log 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38907.log.gz 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25971.log 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48139.log.gz 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42449.log 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51740.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.51740.log.gz 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49071.log 2024-01-12T01:08:19.779 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25971.log.gz 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59923.log 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42449.log.gz 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68891.log 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49071.log.gz 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53724.log 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59923.log: /var/log/ceph/ceph-client.admin.68891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59923.log.gz 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68891.log.gz 2024-01-12T01:08:19.780 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56487.log 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54461.log 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53724.log.gz 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39573.log 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56487.log.gz 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51826.log 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54461.log.gz 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58888.log 2024-01-12T01:08:19.781 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39573.log: 0.0%/var/log/ceph/ceph-client.admin.51826.log: -- replaced with /var/log/ceph/ceph-client.admin.39573.log.gz 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41612.log 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.51826.log.gz 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42191.log 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58888.log.gz 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59725.log 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41612.log.gz 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2024-01-12T01:08:19.782 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.42191.log.gz 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59725.log.gz 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42108.log 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65835.log 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57379.log 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42108.log.gz 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58980.log 2024-01-12T01:08:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65835.log.gz 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69730.log 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.57379.log.gz 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26033.log 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58980.log.gz 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24723.log 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69730.log.gz 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.26033.log: -- /var/log/ceph/ceph-client.admin.37937.log 2024-01-12T01:08:19.784 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26033.log.gz 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40365.log 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24723.log.gz 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60446.log 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37937.log.gz 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33528.log 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40365.log: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.40365.log.gz 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54175.log 2024-01-12T01:08:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60446.log.gz 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32768.log 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33528.log.gz 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54443.log 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42806.log 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54175.log.gz 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39014.log 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32768.log.gz 2024-01-12T01:08:19.786 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54443.log.gz 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63107.log 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42806.log.gz 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36479.log 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39014.log.gz 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69811.log 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63107.log.gz 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55693.log 2024-01-12T01:08:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36479.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36479.log.gz 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39122.log 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69811.log.gz 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39208.log 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55693.log.gz 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25819.log 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39122.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.39122.log.gz--verbose -- 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39594.log 2024-01-12T01:08:19.788 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46061.log 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39208.log.gz 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25819.log.gz 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35639.log 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39594.log.gz 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34027.log 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46061.log.gz 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69568.log 2024-01-12T01:08:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35639.log: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.35639.log.gz 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44185.log 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34027.log.gz 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56530.log 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69568.log.gz 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58750.log 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44185.log.gz 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55026.log 2024-01-12T01:08:19.790 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56530.log.gz 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28474.log 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr: 53.3% -- replaced with /var/log/ceph/ceph-client.admin.58750.log.gz 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25951.log 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55026.log.gz 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32588.log 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28474.log.gz 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51762.log 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25951.log.gz 2024-01-12T01:08:19.791 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35166.log 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32588.log.gz 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40814.log 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51762.log.gz 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57615.log 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35166.log.gz 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31252.log 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40814.log.gz 2024-01-12T01:08:19.792 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34010.log 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57615.log.gz 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48010.log 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31252.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31252.log.gz 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43064.log 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34010.log.gz 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39621.log 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48010.log.gz 2024-01-12T01:08:19.793 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32388.log 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43064.log.gz 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53068.log 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39621.log.gz 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52084.log 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32388.log.gz 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57572.log 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53068.log.gz 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.52084.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44796.log 2024-01-12T01:08:19.794 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52084.log.gz 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46658.log 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57572.log.gz 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31928.log 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44796.log.gz 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65184.log 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46658.log.gz 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55004.log 2024-01-12T01:08:19.795 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31928.log.gz 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52494.log 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65184.log.gz 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58045.log 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55004.log.gz 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64163.log 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52494.log.gz 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64765.log 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58045.log.gz 2024-01-12T01:08:19.796 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31711.log 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64163.log.gz 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34112.log 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64765.log.gz 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34588.log 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31711.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31711.log.gz 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61032.log 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34112.log.gz 2024-01-12T01:08:19.797 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55985.log 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34588.log.gz 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46784.log 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61032.log.gz 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51805.log 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55985.log.gz 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65737.log 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46784.log.gz 2024-01-12T01:08:19.798 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.51805.log: --verbose -- /var/log/ceph/ceph-client.admin.45736.log 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51805.log.gz 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63042.log 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65737.log.gz 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50618.log 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45736.log.gz 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64550.log 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63042.log.gz 2024-01-12T01:08:19.799 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32868.log 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50618.log.gz 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34996.log 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%/var/log/ceph/ceph-client.admin.32868.log: -- replaced with /var/log/ceph/ceph-client.admin.64550.log.gz 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32868.log.gz 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61878.log 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34996.log.gz 2024-01-12T01:08:19.800 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40286.log 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56085.log 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.61878.log.gz 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57916.log 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43519.log 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40286.log: 11.6% -- replaced with /var/log/ceph/ceph-client.admin.56085.log.gz 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60486.log 2024-01-12T01:08:19.801 INFO:teuthology.orchestra.run.smithi178.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.40286.log.gz 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57916.log.gz 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32528.log 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43519.log.gz 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28388.log 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60486.log.gz 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32528.log.gz 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54319.log 2024-01-12T01:08:19.802 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69189.log 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68972.log 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54319.log.gz 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28388.log.gz 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53616.log 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69189.log.gz 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44562.log 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68972.log.gz 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46309.log 2024-01-12T01:08:19.803 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53616.log.gz 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65455.log 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44562.log: /var/log/ceph/ceph-client.admin.46309.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.44562.log.gz 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.46309.log.gz 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57164.log 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55966.log 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65455.log: 12.0% -- replaced with /var/log/ceph/ceph-client.admin.65455.log.gz 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25244.log 2024-01-12T01:08:19.804 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57164.log.gz 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26371.log 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55966.log.gz 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49708.log 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25244.log.gz 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44723.log 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26371.log.gz 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30606.log 2024-01-12T01:08:19.805 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49708.log.gz 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39509.log 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44723.log.gz 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70731.log 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41493.log 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30606.log.gz 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39509.log.gz 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56006.log 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70731.log.gz 2024-01-12T01:08:19.806 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30198.log 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41493.log.gz 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69487.log 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56006.log.gz 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62539.log 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30198.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30198.log.gz 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52343.log 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69487.log.gz 2024-01-12T01:08:19.807 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49600.log 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62539.log.gz 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52343.log: /var/log/ceph/ceph-client.admin.48921.log 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52343.log.gz 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71193.log 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49600.log.gz 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59704.log 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48921.log.gz 2024-01-12T01:08:19.808 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.71193.log: --verbose -- /var/log/ceph/ceph-client.admin.31184.log 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71193.log.gz 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68510.log 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59704.log.gz 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64593.log 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31184.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31184.log.gz 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28517.log 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68510.log.gz 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.64593.log: --verbose -- /var/log/ceph/ceph-client.admin.54501.log 2024-01-12T01:08:19.809 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64593.log.gz 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27422.log 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28517.log.gz 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37236.log 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54501.log.gz 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37765.log 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27422.log: /var/log/ceph/ceph-client.admin.37236.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.27422.log.gz/var/log/ceph/ceph-client.admin.46208.log 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:19.810 INFO:teuthology.orchestra.run.smithi178.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.37236.log.gz 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70866.log 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37765.log.gz 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48203.log 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46208.log.gz 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52795.log 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70866.log.gz 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54521.log 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48203.log.gz 2024-01-12T01:08:19.811 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60116.log 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52795.log.gz 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60972.log 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54521.log.gz 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61461.log 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60116.log.gz 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46870.log 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60972.log.gz 2024-01-12T01:08:19.812 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51783.log 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61461.log.gz 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39229.log 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46870.log: 53.7% -- replaced with /var/log/ceph/ceph-client.admin.46870.log.gz 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40385.log 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51783.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.51783.log.gz 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55931.log 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39229.log.gz 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61133.log 2024-01-12T01:08:19.813 INFO:teuthology.orchestra.run.smithi178.stderr: 53.7% -- replaced with /var/log/ceph/ceph-client.admin.40385.log.gz 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50947.log 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55931.log.gz 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32908.log 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61133.log.gz 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28044.log 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50947.log.gz 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53277.log 2024-01-12T01:08:19.814 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32908.log.gz 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44078.log 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28044.log.gz 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66480.log 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53277.log.gz 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41185.log 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44078.log.gz 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51218.log 2024-01-12T01:08:19.815 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66480.log.gz 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53487.log 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41185.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.41185.log.gz 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65878.log 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51218.log.gz 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.53487.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59332.log 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.53487.log.gz 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34724.log 2024-01-12T01:08:19.816 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65878.log.gz 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39530.log 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr: 51.9% -- replaced with /var/log/ceph/ceph-client.admin.59332.log.gz 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38238.log 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34724.log.gz 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38885.log 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39530.log.gz 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63128.log 2024-01-12T01:08:19.817 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63250.log 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38238.log.gz 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38885.log: gzip /var/log/ceph/ceph-client.admin.63128.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49286.log 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38885.log.gz 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63128.log.gz 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40834.log 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63250.log.gz 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46637.log 2024-01-12T01:08:19.818 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49286.log.gz 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50458.log 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40834.log: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.40834.log.gz 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38023.log 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46637.log.gz 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44142.log 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50458.log.gz 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24788.log 2024-01-12T01:08:19.819 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38023.log.gz 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39272.log 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44142.log.gz 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64185.log 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24788.log.gz 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62388.log 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39272.log.gz 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45312.log 2024-01-12T01:08:19.820 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64185.log.gz 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47687.log 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62388.log.gz 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38718.log 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45312.log.gz 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47687.log.gz 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69757.log 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38718.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.38718.log.gz 2024-01-12T01:08:19.821 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28754.log 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gz 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53767.log 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69757.log: 0.0%/var/log/ceph/ceph-client.admin.28754.log: -- replaced with /var/log/ceph/ceph-client.admin.69757.log.gz 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69406.log 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28754.log.gz 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50288.log 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53767.log.gz 2024-01-12T01:08:19.822 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40425.log 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69406.log.gz 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31507.log 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50288.log.gz 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60721.log 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40425.log: 9.3% -- replaced with /var/log/ceph/ceph-client.admin.40425.log.gz 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60156.log 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31507.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31507.log.gz 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46956.log 2024-01-12T01:08:19.823 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60721.log.gz 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57594.log 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60156.log.gz 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64206.log 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46956.log.gz 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43300.log 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57594.log.gz 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35744.log 2024-01-12T01:08:19.824 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64206.log.gz 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65815.log 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43300.log.gz 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.35744.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.35744.log.gz 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30164.log 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65815.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.65815.log.gz-5 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.68103.log 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-01-12T01:08:19.825 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65985.log 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30164.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30164.log.gz 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45939.log 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68103.log.gz 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32408.log 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65985.log.gz 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70082.log 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45939.log.gz 2024-01-12T01:08:19.826 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27024.log 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32408.log.gz 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30385.log 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70082.log.gz 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31788.log 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27024.log: 3.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.27024.log.gz 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.50519.log 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30385.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30385.log.gz 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50886.log 2024-01-12T01:08:19.827 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31788.log.gz 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69026.log 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50519.log.gz 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38971.log 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50886.log.gz 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61838.log 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69026.log.gz 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62732.log 2024-01-12T01:08:19.828 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38971.log.gz 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33488.log 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61838.log.gz 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35268.log 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62732.log.gz 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30045.log 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33488.log.gz 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63626.log 2024-01-12T01:08:19.829 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35268.log.gz 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62775.log 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30045.log.gz 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28668.log 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63626.log.gz 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25459.log 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62775.log.gz 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57400.log 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28668.log.gz 2024-01-12T01:08:19.830 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54961.log 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25459.log.gz 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46596.log 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57400.log.gz 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54541.log 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54961.log.gz 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47213.log 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46596.log.gz 2024-01-12T01:08:19.831 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32568.log 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54541.log.gz 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25705.log 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47213.log.gz 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47902.log 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32568.log.gz 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25336.log 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25705.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.25705.log.gz 2024-01-12T01:08:19.832 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30317.log 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47902.log.gz 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71166.log 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25336.log.gz 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35353.log 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30317.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30317.log.gz 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71166.log: /var/log/ceph/ceph-client.admin.59964.log 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71166.log.gz 2024-01-12T01:08:19.833 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30368.log 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35353.log.gz 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30691.log 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59964.log.gz 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27400.log 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30368.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30368.log.gz 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30691.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30691.log.gz 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29530.log 2024-01-12T01:08:19.834 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27400.log.gz 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27271.log 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gz 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51136.log 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29530.log.gz 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47385.log 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27271.log.gz 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42491.log 2024-01-12T01:08:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51136.log.gz 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70758.log 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47385.log.gz 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53300.log 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42491.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.42491.log.gz 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70027.log 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70758.log.gz 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53300.log.gz 2024-01-12T01:08:19.836 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55198.log 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70027.log.gz 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28689.log 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60342.log 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55198.log.gz 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49558.log 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28689.log.gz 2024-01-12T01:08:19.837 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31405.log 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60342.log.gz 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65023.log 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49558.log.gz 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29573.log 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31405.log.gz 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65023.log.gz 2024-01-12T01:08:19.838 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56608.log 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29573.log.gz 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38677.log 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42308.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43605.log 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56608.log: 11.6% -- replaced with /var/log/ceph/ceph-client.admin.56608.log.gz 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45183.log 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38677.log.gz 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37194.log 2024-01-12T01:08:19.839 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43605.log.gz 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63755.log 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45183.log.gz 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52192.log 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37194.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.37194.log.gz 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49794.log 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63755.log.gz 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63841.log 2024-01-12T01:08:19.840 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52192.log.gz 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26963.log 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49794.log.gz 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57078.log 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63841.log.gz 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58024.log 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26963.log.gz 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37488.log 2024-01-12T01:08:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57078.log.gz 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38560.log 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58024.log.gz 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30674.log 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37488.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.37488.log.gz 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59373.log 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38560.log.gz 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25497.log 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30674.log.gz 2024-01-12T01:08:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48684.log 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49050.log 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59373.log.gz 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43970.log /var/log/ceph/ceph-client.admin.48684.log: 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25497.log.gz 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48684.log.gz 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24046.log 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49050.log.gz 2024-01-12T01:08:19.843 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27872.log 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43970.log.gz 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65899.log 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24046.log.gz 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24874.log 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27872.log.gz 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65331.log 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65899.log.gz 2024-01-12T01:08:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55090.log 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24874.log.gz 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51697.log 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65331.log.gz 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23588.log 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55090.log.gz 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38066.log 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51697.log.gz 2024-01-12T01:08:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56626.log 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23588.log.gz 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38066.log.gz 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68402.log 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56626.log.gz 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52838.log 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41591.log: 23.5%/var/log/ceph/ceph-client.admin.68402.log: -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-01-12T01:08:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52644.log 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68402.log.gz 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24465.log 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52838.log.gz 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55769.log 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52644.log.gz 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24895.log 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24465.log.gz 2024-01-12T01:08:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30436.log 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55769.log.gz 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24895.log.gz 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58959.log 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30436.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30436.log.gz 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44099.log 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2024-01-12T01:08:19.848 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65311.log 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58959.log.gz 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64077.log 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44099.log.gz 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25024.log 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65311.log.gz 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52170.log 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64077.log.gz 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25024.log: /var/log/ceph/ceph-client.admin.57207.log 2024-01-12T01:08:19.849 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25024.log.gz 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63604.log 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52170.log.gz 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61759.log 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57207.log.gz 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30487.log 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.63604.log.gz 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36711.log 2024-01-12T01:08:19.850 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61759.log.gz 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32828.log 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30487.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30487.log.gz 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45575.log 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36711.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36711.log.gz 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32828.log.gz 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31541.log 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45575.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.45575.log.gzgzip 2024-01-12T01:08:19.851 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.70217.log 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36269.log 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31541.log.gz 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48515.log 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70217.log.gz 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68672.log 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36269.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.36269.log.gz 2024-01-12T01:08:19.852 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49093.log 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48515.log.gz 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38430.log 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68672.log.gz 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50659.log 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49093.log.gz 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48241.log 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50418.log 2024-01-12T01:08:19.853 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.38430.log.gz 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50659.log.gz 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37215.log 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48241.log.gz 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52537.log 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50418.log.gz 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43669.log 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.37215.log.gz 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56323.log 2024-01-12T01:08:19.854 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52537.log.gz 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26328.log 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43669.log.gz 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66157.log 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56323.log: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.56323.log.gz 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68837.log 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26328.log.gz 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63948.log 2024-01-12T01:08:19.855 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66157.log.gz 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68321.log 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68837.log.gz 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60799.log 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63948.log.gz 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.68321.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51633.log 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68321.log.gz 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40086.log 2024-01-12T01:08:19.856 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60799.log.gz 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49901.log 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51633.log.gz 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53132.log 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40086.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.40086.log.gz 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20053.log 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49901.log.gz 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55112.log 2024-01-12T01:08:19.857 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53132.log.gz 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.20053.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26115.log 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20053.log.gz 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24616.log 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55112.log.gz 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46414.log 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26115.log.gz 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27680.log 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24616.log.gz 2024-01-12T01:08:19.858 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26629.log 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46414.log.gz 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61538.log 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27680.log.gz 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31490.log 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26629.log.gz 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66179.log 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61538.log.gz 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67615.log 2024-01-12T01:08:19.859 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31490.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31490.log.gz 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35555.log 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66179.log.gz 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40485.log 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67615.log.gz 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65964.log 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35555.log: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.35555.log.gz 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55575.log 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40485.log: 16.4% -- replaced with /var/log/ceph/ceph-client.admin.40485.log.gz 2024-01-12T01:08:19.860 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66761.log 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65964.log.gz 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45798.log 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55575.log.gz 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42370.log 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66761.log.gz 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36858.log 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45798.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.45798.log.gz 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38601.log 2024-01-12T01:08:19.861 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42370.log.gz 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54282.log 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36858.log: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36858.log.gz 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27658.log 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38601.log.gz 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58905.log 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54282.log.gz 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43171.log 2024-01-12T01:08:19.862 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27658.log.gz 2024-01-12T01:08:19.863 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51998.log 2024-01-12T01:08:19.863 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58905.log.gz 2024-01-12T01:08:19.863 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.43171.log: -- /var/log/ceph/ceph-client.admin.65617.log 2024-01-12T01:08:19.863 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43171.log.gz 2024-01-12T01:08:19.863 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66995.log 2024-01-12T01:08:19.863 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51998.log.gz 2024-01-12T01:08:19.863 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65617.log.gz 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40794.log 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66995.log.gz 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70596.log 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71302.log 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40794.log.gz 2024-01-12T01:08:19.864 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41311.log 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70596.log.gz 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34078.log 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71302.log.gz 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32548.log 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41311.log: gzip -5 23.7%--verbose -- replaced with /var/log/ceph/ceph-client.admin.41311.log.gz-- 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43386.log 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34078.log.gz 2024-01-12T01:08:19.865 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30113.log 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32548.log.gz 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24230.log 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43386.log.gz 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44369.log 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30113.log.gz 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35421.log 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24230.log.gz 2024-01-12T01:08:19.866 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40997.log 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44369.log.gz 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49222.log 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35421.log.gz 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.40997.log: --verbose -- /var/log/ceph/ceph-client.admin.29358.log 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.40997.log.gz 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64980.log 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49222.log.gz 2024-01-12T01:08:19.867 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59022.log 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29358.log.gz 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67943.log 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64980.log.gz 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30096.log 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59022.log.gz 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36017.log 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67943.log.gz 2024-01-12T01:08:19.868 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64056.log 2024-01-12T01:08:19.869 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30096.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30096.log.gz 2024-01-12T01:08:19.869 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24126.log 2024-01-12T01:08:19.869 INFO:teuthology.orchestra.run.smithi178.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36017.log.gz 2024-01-12T01:08:19.869 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62517.log 2024-01-12T01:08:19.869 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64056.log.gz 2024-01-12T01:08:19.869 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59139.log 2024-01-12T01:08:19.869 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24126.log.gz 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59430.log /var/log/ceph/ceph-client.admin.62517.log: 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62517.log.gz 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39100.log 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59139.log.gz 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46763.log 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59430.log.gz 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66458.log 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39100.log.gz 2024-01-12T01:08:19.870 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46849.log 2024-01-12T01:08:19.871 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46763.log.gz 2024-01-12T01:08:19.871 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49136.log 2024-01-12T01:08:19.871 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66458.log.gz 2024-01-12T01:08:19.871 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38259.log 2024-01-12T01:08:19.871 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46849.log.gz 2024-01-12T01:08:19.871 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59451.log 2024-01-12T01:08:19.871 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49136.log.gz 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54366.log 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38259.log.gz 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64529.log 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59451.log.gz 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27529.log 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32008.log 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54366.log.gz 2024-01-12T01:08:19.872 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64529.log.gz 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56124.log 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27529.log.gz 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55630.log 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32008.log.gz 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24766.log 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56124.log.gz 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55630.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.58153.log -- replaced with /var/log/ceph/ceph-client.admin.55630.log.gz 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40566.log 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24766.log.gz 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24422.log 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58153.log: /var/log/ceph/ceph-client.admin.40566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58153.log.gz 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54768.log 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40566.log.gz 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56282.log 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24422.log.gz 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29659.log 2024-01-12T01:08:19.874 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54768.log.gz 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47149.log 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56282.log.gz 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29001.log 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29659.log.gz 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30708.log 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47149.log.gz 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66672.log 2024-01-12T01:08:19.875 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29001.log.gz 2024-01-12T01:08:19.876 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57787.log 2024-01-12T01:08:19.876 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30708.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30708.log.gz 2024-01-12T01:08:19.876 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33891.log 2024-01-12T01:08:19.876 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66672.log: 54.3% -- replaced with /var/log/ceph/ceph-client.admin.66672.log.gz 2024-01-12T01:08:19.876 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25439.log 2024-01-12T01:08:19.876 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57787.log.gz 2024-01-12T01:08:19.876 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55593.log 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33891.log.gz 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28001.log 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25439.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25439.log.gz 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49751.log 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55593.log.gz 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30759.log 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28001.log.gz 2024-01-12T01:08:19.877 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.49751.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38281.log 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49751.log.gz 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48389.log 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60895.log 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30759.log.gz 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38281.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38281.log.gz-5 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.32468.log 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48389.log.gz 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28302.log 2024-01-12T01:08:19.878 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60895.log.gz 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48556.log 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32468.log.gz 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27045.log 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28302.log.gz 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47428.log 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48556.log.gz 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26693.log 2024-01-12T01:08:19.879 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27045.log.gz 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.47428.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36837.log 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47428.log.gz 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52408.log 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26693.log.gz 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30351.log 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36837.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36837.log.gz 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38467.log 2024-01-12T01:08:19.880 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52408.log.gz 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31694.log 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30351.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30351.log.gz 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33028.log 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38467.log.gz 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62013.log 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31694.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31694.log.gz 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60302.log 2024-01-12T01:08:19.881 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33028.log.gz 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32428.log 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62013.log: 53.7% -- replaced with /var/log/ceph/ceph-client.admin.62013.log.gz 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47342.log 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60302.log.gz 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.32428.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51260.log 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32428.log.gz 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48641.log 2024-01-12T01:08:19.882 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47342.log.gz 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35870.log 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51260.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.51260.log.gz 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33088.log 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48641.log.gz 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45527.log 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58791.log 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.35870.log.gz 2024-01-12T01:08:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33088.log.gz 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45032.log 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45527.log.gz 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59848.log 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58791.log.gz 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69053.log /var/log/ceph/ceph-client.admin.45032.log: 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45032.log.gz 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66973.log 2024-01-12T01:08:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59848.log.gz 2024-01-12T01:08:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25226.log 2024-01-12T01:08:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69053.log.gz 2024-01-12T01:08:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60526.log 2024-01-12T01:08:19.885 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66973.log.gz 2024-01-12T01:08:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53447.log 2024-01-12T01:08:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64701.log 2024-01-12T01:08:19.885 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25226.log.gz 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60526.log.gz 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68756.log 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53447.log.gz 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26478.log 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64701.log.gz 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60661.log 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68756.log.gz 2024-01-12T01:08:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24981.log 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26478.log.gz 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36416.log 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60661.log.gz 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34860.log 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24981.log.gz 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36732.log 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36416.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.36416.log.gz 2024-01-12T01:08:19.887 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63291.log 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34860.log.gz 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42978.log 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36732.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.36732.log.gz 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44989.log 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63291.log: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.63291.log.gz 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.42978.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67703.log 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42978.log.gz 2024-01-12T01:08:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.44989.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65144.log 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44989.log.gz 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28109.log 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67703.log.gz 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35387.log 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65144.log.gz 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34095.log 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28109.log.gz 2024-01-12T01:08:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35387.log.gz 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29766.log 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35183.log 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34095.log.gz 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28625.log 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29766.log.gz 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24444.log 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35183.log.gz 2024-01-12T01:08:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45247.log 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28625.log.gz 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.24444.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70406.log 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24444.log.gz 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25046.log 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45247.log.gz 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47580.log 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70406.log.gz 2024-01-12T01:08:19.891 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27786.log 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25046.log.gz 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31677.log 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47580.log: /var/log/ceph/ceph-client.admin.27786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47580.log.gz 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47859.log 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.27786.log.gz 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31677.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31677.log.gz 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58922.log 2024-01-12T01:08:19.892 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47859.log.gz 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41041.log /var/log/ceph/ceph-mgr.x.log: 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41393.log 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58922.log.gz 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58195.log 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41041.log.gz 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59354.log 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41393.log.gz 2024-01-12T01:08:19.893 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26457.log 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58195.log.gz 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67683.log 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59354.log.gz 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27144.log 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26457.log.gz 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63392.log 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67683.log.gz 2024-01-12T01:08:19.894 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33108.log 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27144.log.gz 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36248.log /var/log/ceph/ceph-client.admin.63392.log: 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63392.log.gz 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47020.log 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33108.log.gz 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45778.log 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36248.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.36248.log.gz 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58248.log 2024-01-12T01:08:19.895 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39057.log 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45778.log.gz 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36963.log 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58248.log.gz 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26822.log 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39057.log.gz 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24551.log 2024-01-12T01:08:19.896 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36963.log.gz 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66286.log 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26822.log.gz 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24551.log.gz 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59392.log 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44620.log 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29702.log 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66286.log.gz 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59392.log: /var/log/ceph/ceph-client.admin.44620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59392.log.gz 2024-01-12T01:08:19.897 INFO:teuthology.orchestra.run.smithi178.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44620.log.gz 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.41987.log 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25199.log 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29702.log.gz 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41145.log 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41987.log.gz 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67554.log 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25199.log.gz 2024-01-12T01:08:19.898 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54240.log 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41145.log.gz 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.67554.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24204.log 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67554.log.gz 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.54240.log: --verbose -- /var/log/ceph/ceph-client.admin.33806.log 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54240.log.gz 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57056.log 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24204.log.gz 2024-01-12T01:08:19.899 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2024-01-12T01:08:19.900 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33806.log.gz 2024-01-12T01:08:19.900 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42470.log 2024-01-12T01:08:19.900 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57056.log.gz 2024-01-12T01:08:19.900 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59884.log 2024-01-12T01:08:19.900 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2024-01-12T01:08:19.900 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31764.log 2024-01-12T01:08:19.900 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42470.log.gz 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62054.log 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59884.log.gz 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51339.log 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46721.log 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr: 54.7% -- replaced with /var/log/ceph/ceph-client.admin.31764.log.gz 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62054.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.62054.log.gz 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.46164.log 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51339.log.gz 2024-01-12T01:08:19.901 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58366.log 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46721.log.gz 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65697.log 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30980.log 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58366.log.gz 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52020.log 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr: 11.1% -- replaced with /var/log/ceph/ceph-client.admin.65697.log.gz 2024-01-12T01:08:19.902 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.30980.log: --verbose -- /var/log/ceph/ceph-client.admin.44682.log 2024-01-12T01:08:19.903 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30980.log.gz 2024-01-12T01:08:19.903 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27980.log 2024-01-12T01:08:19.903 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52020.log.gz 2024-01-12T01:08:19.903 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31968.log 2024-01-12T01:08:19.903 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44682.log.gz 2024-01-12T01:08:19.903 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54789.log 2024-01-12T01:08:19.903 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27980.log.gz 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61698.log 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31968.log.gz 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31888.log 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54789.log.gz 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51092.log 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61698.log: 27.9% -- replaced with /var/log/ceph/ceph-client.admin.61698.log.gz 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34265.log 2024-01-12T01:08:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31888.log.gz 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60242.log 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51092.log.gz 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46556.log 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34265.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.34265.log.gz-- 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43476.log 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25857.log 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60242.log.gz 2024-01-12T01:08:19.905 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46556.log.gz 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40141.log 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43476.log.gz 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33468.log 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25857.log.gz 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64744.log 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.40141.log.gz 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29250.log 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33468.log.gz 2024-01-12T01:08:19.906 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23668.log 2024-01-12T01:08:19.907 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64744.log.gz 2024-01-12T01:08:19.907 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68051.log 2024-01-12T01:08:19.907 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29250.log.gz 2024-01-12T01:08:19.907 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43021.log 2024-01-12T01:08:19.907 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23668.log.gz 2024-01-12T01:08:19.907 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48492.log 2024-01-12T01:08:19.907 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68051.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.68051.log.gz--verbose 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.57981.log 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43021.log.gz 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36900.log 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48492.log.gz 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71004.log 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57981.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.57981.log.gz-5 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.65104.log 2024-01-12T01:08:19.908 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36900.log: /var/log/ceph/ceph-client.admin.71004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43949.log 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36900.log.gz 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71004.log.gz 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52945.log 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65104.log.gz 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63712.log 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43949.log.gz 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25414.log 2024-01-12T01:08:19.909 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52945.log.gz 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31473.log 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63712.log.gz 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25394.log 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25414.log.gz 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28087.log 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31473.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31473.log.gz 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47752.log 2024-01-12T01:08:19.910 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25394.log.gz 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57228.log 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28087.log.gz 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55672.log 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47752.log.gz 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31948.log 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57228.log.gz 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68537.log 2024-01-12T01:08:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55672.log.gz 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63332.log 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31948.log.gz 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32368.log 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68537.log.gz 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49494.log 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63332.log.gz 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41950.log 2024-01-12T01:08:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32368.log.gz 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45118.log 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr: 55.3% -- replaced with /var/log/ceph/ceph-client.admin.49494.log.gz 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.41950.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35975.log 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41950.log.gz 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50844.log 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45118.log.gz 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-12T01:08:19.913 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35975.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.35975.log.gz 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50722.log 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50844.log.gz 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30929.log 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.50722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33168.log 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50722.log.gz 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.30929.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43150.log 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30929.log.gz 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41410.log 2024-01-12T01:08:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33168.log.gz 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59507.log 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43150.log.gz 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54403.log 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41410.log.gz 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52149.log 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59507.log.gz 2024-01-12T01:08:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53788.log 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54403.log.gz 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49114.log 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52149.log.gz 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29852.log 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53788.log.gz 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49114.log: /var/log/ceph/ceph-client.admin.31371.log 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49114.log.gz 2024-01-12T01:08:19.916 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53681.log 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29852.log: 23.3%/var/log/ceph/ceph-client.admin.31371.log: -- replaced with /var/log/ceph/ceph-client.admin.29852.log.gz 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45462.log 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31371.log.gz 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34911.log 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53681.log.gz 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62668.log 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45462.log.gz 2024-01-12T01:08:19.917 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.34911.log: -- /var/log/ceph/ceph-client.admin.48471.log 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34911.log.gz 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55730.log 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62668.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.62668.log.gz--verbose 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.53384.log 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48471.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz-5 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.25418.log 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55730.log.gz 2024-01-12T01:08:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57852.log 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53384.log.gz 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50926.log 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25418.log.gz 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40708.log 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57852.log.gz 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63690.log 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50926.log.gz 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54197.log 2024-01-12T01:08:19.919 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40708.log.gz 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63690.log.gz 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59470.log 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54197.log.gz 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69135.log 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54421.log 2024-01-12T01:08:19.920 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59470.log.gz 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19870.log 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69135.log.gz 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28984.log 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54421.log.gz 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24178.log 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.19870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19870.log.gz 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28984.log: /var/log/ceph/ceph-client.admin.41225.log 2024-01-12T01:08:19.921 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28984.log.gz 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48096.log 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24178.log.gz 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27224.log 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41225.log: 55.4% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56906.log 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48096.log.gz 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50680.log 2024-01-12T01:08:19.922 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27224.log.gz 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49944.log 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52580.log 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56906.log.gz 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33668.log 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50680.log.gz 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49944.log.gz 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.52580.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-01-12T01:08:19.923 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52580.log.gz 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57830.log 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33668.log.gz 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30300.log 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24659.log 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57830.log.gz 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2024-01-12T01:08:19.924 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30300.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30300.log.gz 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42956.log 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24659.log.gz 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24637.log 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69649.log 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42956.log.gz 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61973.log 2024-01-12T01:08:19.925 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24637.log.gz 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57873.log 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69649.log.gz 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70923.log 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61973.log.gz 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.57873.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58400.log 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57873.log.gz 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56407.log 2024-01-12T01:08:19.926 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70923.log.gz 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70136.log 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58400.log.gz 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24100.log 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56407.log.gz 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60992.log 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70136.log.gz 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50331.log 2024-01-12T01:08:19.927 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24100.log.gz 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64443.log 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60992.log.gz 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26156.log 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50331.log.gz 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51176.log 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64443.log.gz 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70352.log 2024-01-12T01:08:19.928 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26156.log.gz 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48620.log 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51176.log.gz 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68618.log 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70352.log.gz 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35404.log 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48620.log.gz 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61819.log 2024-01-12T01:08:19.929 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68618.log.gz 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35404.log.gz 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62453.log 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48835.log 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61819.log.gz 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25725.log 2024-01-12T01:08:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62453.log.gz 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49308.log 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48835.log.gz 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63884.log 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25725.log.gz 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45333.log 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49308.log.gz 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51891.log 2024-01-12T01:08:19.931 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63884.log.gz 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32128.log 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45333.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45333.log.gz/var/log/ceph/ceph-client.admin.63230.log 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51891.log.gz 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46143.log 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32128.log.gz 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52213.log 2024-01-12T01:08:19.932 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63230.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63230.log.gz 2024-01-12T01:08:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.37257.log 2024-01-12T01:08:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46143.log.gz 2024-01-12T01:08:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67722.log 2024-01-12T01:08:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52213.log.gz 2024-01-12T01:08:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24072.log 2024-01-12T01:08:19.933 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37257.log.gz 2024-01-12T01:08:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40225.log 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67722.log: /var/log/ceph/ceph-client.admin.24072.log: 79.3% -- replaced with /var/log/ceph/ceph-client.admin.67722.log.gz 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64507.log 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24072.log.gz 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52752.log 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40225.log: /var/log/ceph/ceph-client.admin.64507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24401.log 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64507.log.gz 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.40225.log.gz 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33588.log 2024-01-12T01:08:19.934 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52752.log.gz 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48327.log 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24401.log.gz 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70000.log 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33588.log.gz 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63434.log 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48327.log.gz 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42892.log 2024-01-12T01:08:19.935 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70000.log.gz 2024-01-12T01:08:19.936 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49375.log 2024-01-12T01:08:19.936 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63434.log.gz 2024-01-12T01:08:19.936 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25174.log 2024-01-12T01:08:19.936 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42892.log.gz 2024-01-12T01:08:19.936 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35234.log 2024-01-12T01:08:19.936 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49375.log.gz 2024-01-12T01:08:19.936 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42088.log 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25174.log.gz 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38302.log /var/log/ceph/ceph-client.admin.35234.log: 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35234.log.gz 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.42088.log: --verbose -- /var/log/ceph/ceph-client.admin.30283.log 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.42088.log.gz 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52386.log 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38302.log.gz 2024-01-12T01:08:19.937 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.30283.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33688.log 2024-01-12T01:08:19.938 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30283.log.gz 2024-01-12T01:08:19.938 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57486.log 2024-01-12T01:08:19.938 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52386.log.gz 2024-01-12T01:08:19.938 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31908.log 2024-01-12T01:08:19.938 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33688.log.gz 2024-01-12T01:08:19.938 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46123.log 2024-01-12T01:08:19.938 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57486.log.gz 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47838.log 0.0% 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31908.log.gz 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.46123.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43734.log 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46123.log.gz 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48451.log 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47838.log.gz 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48305.log 2024-01-12T01:08:19.939 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43734.log.gz 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30776.log 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48451.log.gz 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55831.log 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48305.log.gz 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30776.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30776.log.gz 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68483.log 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55831.log.gz 2024-01-12T01:08:19.940 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47084.log 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40183.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64335.log 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68483.log.gz 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47084.log.gz 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62173.log 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64335.log.gz 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29422.log 2024-01-12T01:08:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41888.log: gzip -5 24.1%--verbose -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz-- 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58519.log 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62173.log.gz 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48577.log 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29422.log.gz 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56863.log 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58519.log.gz 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25553.log 2024-01-12T01:08:19.942 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48577.log.gz 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36311.log 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33008.log 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56863.log.gz 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44334.log 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36311.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25553.log.gz 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.36311.log.gz 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58110.log 2024-01-12T01:08:19.943 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33008.log.gz 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35912.log 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44334.log.gz 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62942.log 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58110.log.gz 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64937.log 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35912.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.35912.log.gz 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42512.log 2024-01-12T01:08:19.944 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62942.log.gz 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55176.log 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64937.log.gz 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61738.log 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42512.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.42512.log.gz-- 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38739.log 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55176.log.gz 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70109.log 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61738.log.gz 2024-01-12T01:08:19.945 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27551.log 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66200.log 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70109.log.gz 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30742.log 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54854.log 0.0% 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27551.log.gz 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36753.log 2024-01-12T01:08:19.946 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66200.log.gz 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30742.log.gz 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71274.log 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54854.log.gz 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60778.log 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.36753.log.gz 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41810.log 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71274.log.gz 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.60778.log: -- /var/log/ceph/ceph-client.admin.32208.log 2024-01-12T01:08:19.947 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60778.log.gz 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63457.log 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41810.log.gz 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.32208.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37958.log 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32208.log.gz 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70054.log 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63457.log.gz 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-01-12T01:08:19.948 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37958.log.gz 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36122.log 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70054.log.gz 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42655.log 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65597.log 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36122.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36122.log.gz 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57701.log 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42655.log.gz 2024-01-12T01:08:19.949 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71112.log 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45979.log 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65597.log.gz 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57701.log: /var/log/ceph/ceph-client.admin.71112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46435.log 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57701.log.gz 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71112.log.gz 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70460.log 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45979.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45979.log.gz 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.69243.log 2024-01-12T01:08:19.950 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46435.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46435.log.gz-5 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.67253.log 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70460.log.gz 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33128.log 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69243.log.gz 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29174.log 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67253.log.gz 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38087.log 2024-01-12T01:08:19.951 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33128.log.gz 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29174.log.gz 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26983.log 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38087.log.gz 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67916.log 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32148.log 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr: 5.3% -- replaced with /var/log/ceph/ceph-client.admin.26983.log.gz 2024-01-12T01:08:19.952 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67916.log.gz 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57314.log 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32148.log: /var/log/ceph/ceph-client.admin.42291.log 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32148.log.gz 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52902.log 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57314.log.gz 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71247.log 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42291.log.gz 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58869.log 2024-01-12T01:08:19.953 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52902.log.gz 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63002.log 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71247.log.gz 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46827.log 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58869.log.gz 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38486.log 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63002.log.gz 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37446.log 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46827.log.gz 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-01-12T01:08:19.954 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38486.log.gz 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65794.log 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37446.log: /var/log/ceph/ceph-client.admin.43583.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.37446.log.gz 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58668.log 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47235.log 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59684.log 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr: 52.9% -- replaced with /var/log/ceph/ceph-client.admin.65794.log.gz 2024-01-12T01:08:19.955 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57809.log 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58668.log.gz 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47235.log.gz 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25667.log 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59684.log.gz 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66844.log 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57809.log.gz 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44946.log 2024-01-12T01:08:19.956 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25667.log.gz 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35336.log 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66844.log.gz 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58214.log 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44946.log.gz 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63372.log 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35336.log.gz 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30963.log 2024-01-12T01:08:19.957 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58214.log.gz 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23992.log 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63372.log: gzip /var/log/ceph/ceph-client.admin.30963.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50541.log 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.63372.log.gz 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30963.log.gz 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59082.log 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29487.log 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23992.log.gz 2024-01-12T01:08:19.958 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50541.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50541.log.gz 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.59230.log 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59082.log.gz 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29487.log.gz 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42270.log 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59230.log.gz 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57766.log 2024-01-12T01:08:19.959 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44210.log: /var/log/ceph/ceph-client.admin.42270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-01-12T01:08:19.960 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38524.log 2024-01-12T01:08:19.960 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.42270.log.gz 2024-01-12T01:08:19.960 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43562.log 2024-01-12T01:08:19.960 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57766.log.gz 2024-01-12T01:08:19.960 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31354.log 2024-01-12T01:08:19.960 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38524.log.gz 2024-01-12T01:08:19.960 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41967.log 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43562.log.gz/var/log/ceph/ceph-client.admin.31354.log: 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26779.log 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31354.log.gz 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66501.log 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41967.log: /var/log/ceph/ceph-client.admin.26779.log: gzip 23.4% -- replaced with /var/log/ceph/ceph-client.admin.41967.log.gz-5 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.39143.log 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26779.log.gz 2024-01-12T01:08:19.961 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27958.log 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66501.log.gz 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28866.log 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39143.log.gz 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48985.log 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36395.log 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27958.log.gz 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54875.log 2024-01-12T01:08:19.962 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28866.log.gz 2024-01-12T01:08:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48985.log.gz 2024-01-12T01:08:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47451.log 2024-01-12T01:08:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36395.log: /var/log/ceph/ceph-client.admin.54875.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36395.log.gz 2024-01-12T01:08:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55870.log 2024-01-12T01:08:19.963 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54875.log.gz 2024-01-12T01:08:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29077.log 2024-01-12T01:08:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47451.log.gzgzip 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.61229.log 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64292.log 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55870.log.gz 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29077.log.gz 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62797.log 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61229.log.gz 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55849.log 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64292.log.gz 2024-01-12T01:08:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24152.log 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62797.log.gz 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57465.log 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55849.log.gz 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63970.log 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24152.log.gz 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47106.log 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57465.log.gz 2024-01-12T01:08:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.63970.log: -- /var/log/ceph/ceph-client.admin.67210.log 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63970.log.gz 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66329.log 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47106.log.gz 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24530.log 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67210.log.gz 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45815.log 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66329.log.gz 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27894.log 2024-01-12T01:08:19.966 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24530.log.gz 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42050.log 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45815.log.gz 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54983.log 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27894.log.gz 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44035.log 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.42050.log.gz 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59160.log 2024-01-12T01:08:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54983.log.gz 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35660.log 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44035.log.gz 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.59160.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67446.log 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59160.log.gz 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34639.log 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.35660.log.gz 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29272.log 2024-01-12T01:08:19.968 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67446.log.gz 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58810.log 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34639.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34639.log.gz-5 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.51505.log 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29272.log.gz 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51281.log 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58810.log.gz 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49419.log 2024-01-12T01:08:19.969 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51505.log.gz 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32188.log 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51281.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.51281.log.gz 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66437.log 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49419.log.gz 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64034.log 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32188.log.gz 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71220.log 2024-01-12T01:08:19.970 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66437.log.gz 2024-01-12T01:08:19.971 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55284.log 2024-01-12T01:08:19.971 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64034.log.gz 2024-01-12T01:08:19.971 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32068.log 2024-01-12T01:08:19.971 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71220.log.gz 2024-01-12T01:08:19.971 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31150.log 2024-01-12T01:08:19.971 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30912.log 2024-01-12T01:08:19.971 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55284.log.gz 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40266.log 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32068.log.gz 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31150.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31150.log.gz 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38324.log 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30912.log.gz 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55305.log 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr: 56.2% -- replaced with /var/log/ceph/ceph-client.admin.40266.log.gz 2024-01-12T01:08:19.972 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46040.log 0.0% 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38324.log.gz 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55305.log.gz 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.42741.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63927.log 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31422.log 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46040.log.gz 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57959.log 2024-01-12T01:08:19.973 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63927.log.gz 2024-01-12T01:08:19.974 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43279.log 2024-01-12T01:08:19.974 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31422.log.gz 2024-01-12T01:08:19.974 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53745.log 2024-01-12T01:08:19.974 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57959.log.gz 2024-01-12T01:08:19.974 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.43279.log: --verbose -- /var/log/ceph/ceph-client.admin.53154.log 2024-01-12T01:08:19.974 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43279.log.gz 2024-01-12T01:08:19.974 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43540.log 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53745.log.gz 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.53154.log: -- /var/log/ceph/ceph-client.admin.52881.log 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53154.log.gz 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36206.log 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43540.log.gz 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48430.log 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52881.log.gz 2024-01-12T01:08:19.975 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28603.log 2024-01-12T01:08:19.976 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36206.log.gz 2024-01-12T01:08:19.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66093.log 2024-01-12T01:08:19.976 INFO:teuthology.orchestra.run.smithi178.stderr: 26.5% -- replaced with /var/log/ceph/ceph-client.admin.48430.log.gz 2024-01-12T01:08:19.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31014.log 2024-01-12T01:08:19.976 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28603.log.gz 2024-01-12T01:08:19.976 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35319.log 2024-01-12T01:08:19.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66093.log.gz 2024-01-12T01:08:19.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29379.log 2024-01-12T01:08:19.977 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31014.log.gz 2024-01-12T01:08:19.977 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32308.log 2024-01-12T01:08:19.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35319.log.gz 2024-01-12T01:08:19.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68591.log 2024-01-12T01:08:19.977 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29379.log.gz 2024-01-12T01:08:19.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66028.log 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32308.log.gz 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31116.log 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68591.log.gz 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69541.log 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66028.log.gz 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40162.log 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31116.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31116.log.gz 2024-01-12T01:08:19.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24372.log 2024-01-12T01:08:19.979 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69541.log.gz 2024-01-12T01:08:19.979 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37679.log 2024-01-12T01:08:19.979 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40162.log.gz 2024-01-12T01:08:19.979 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48727.log 2024-01-12T01:08:19.979 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24372.log.gz 2024-01-12T01:08:19.979 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58731.log 2024-01-12T01:08:19.979 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37679.log.gz 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43712.log 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48727.log.gz 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62345.log 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58731.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.58731.log.gz 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44298.log 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43712.log.gz 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47623.log /var/log/ceph/ceph-client.admin.62345.log: 2024-01-12T01:08:19.980 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62345.log.gz 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55910.log 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42612.log 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44298.log.gz 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47623.log: /var/log/ceph/ceph-client.admin.55910.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.54681.log 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47623.log.gz 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55910.log.gz 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45204.log 2024-01-12T01:08:19.981 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.42612.log.gz 2024-01-12T01:08:19.982 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64851.log 2024-01-12T01:08:19.982 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54681.log.gz 2024-01-12T01:08:19.982 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36164.log 2024-01-12T01:08:19.982 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45204.log.gz 2024-01-12T01:08:19.982 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24508.log 2024-01-12T01:08:19.982 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64851.log.gz 2024-01-12T01:08:19.982 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63170.log 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36164.log: /var/log/ceph/ceph-client.admin.24508.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36164.log.gz 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.24508.log.gz 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61193.log 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46269.log 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63170.log.gz 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36521.log 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61193.log.gz 2024-01-12T01:08:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47730.log 2024-01-12T01:08:19.984 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46269.log.gz 2024-01-12T01:08:19.984 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30538.log 2024-01-12T01:08:19.984 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36521.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.36521.log.gz 2024-01-12T01:08:19.984 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57271.log 2024-01-12T01:08:19.984 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47730.log.gz 2024-01-12T01:08:19.984 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34792.log 2024-01-12T01:08:19.984 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30538.log.gz 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37980.log 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57271.log.gz 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67575.log 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34792.log.gz 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39945.log 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37980.log.gz 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61678.log 2024-01-12T01:08:19.985 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67575.log.gz 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67705.log 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39945.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.39945.log.gz 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32088.log 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61678.log.gz 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33959.log 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67705.log.gz 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49729.log 2024-01-12T01:08:19.986 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32088.log.gz 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33959.log.gz 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40465.log 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49729.log.gz 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51072.log 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29959.log 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr: 58.1% -- replaced with /var/log/ceph/ceph-client.admin.40465.log.gz 2024-01-12T01:08:19.987 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34571.log 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51072.log.gz 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41633.log 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29959.log.gz 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19994.log 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34571.log.gz 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40307.log 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-01-12T01:08:19.988 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.19994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19994.log.gz 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69595.log 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40307.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.40307.log.gz 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47816.log 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44903.log 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69595.log.gz 2024-01-12T01:08:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61937.log 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47816.log.gz 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62982.log 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44903.log.gz 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.61937.log: --verbose -- /var/log/ceph/ceph-client.admin.61798.log 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61937.log.gz 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60222.log 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62982.log.gz 2024-01-12T01:08:19.990 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49815.log 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61798.log: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.61798.log.gz 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60222.log.gz 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69703.log 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49815.log.gz 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45054.log 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25003.log 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-01-12T01:08:19.991 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57185.log 2024-01-12T01:08:19.992 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69703.log.gz 2024-01-12T01:08:19.992 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45054.log.gz 2024-01-12T01:08:19.992 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34316.log 2024-01-12T01:08:19.992 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25003.log.gz 2024-01-12T01:08:19.992 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38779.log 2024-01-12T01:08:19.992 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57185.log.gz 2024-01-12T01:08:19.992 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31524.log 2024-01-12T01:08:19.993 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34316.log.gz 2024-01-12T01:08:19.993 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.38779.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59807.log 2024-01-12T01:08:19.993 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38779.log.gz 2024-01-12T01:08:19.993 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68945.log 2024-01-12T01:08:19.993 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31524.log.gz 2024-01-12T01:08:19.993 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56734.log 2024-01-12T01:08:19.993 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24487.log 2024-01-12T01:08:19.994 INFO:teuthology.orchestra.run.smithi178.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.59807.log.gz 2024-01-12T01:08:19.994 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68945.log.gz 2024-01-12T01:08:19.994 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71329.log 2024-01-12T01:08:19.994 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57013.log 2024-01-12T01:08:19.994 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56734.log.gz 2024-01-12T01:08:19.994 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24487.log.gz 2024-01-12T01:08:19.994 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51441.log 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71329.log.gz 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35786.log 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57013.log.gz 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59664.log 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51441.log.gz 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69297.log 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.35786.log.gz 2024-01-12T01:08:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25110.log 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59664.log.gz 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37425.log 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69297.log.gz 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27701.log 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25110.log.gz 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43626.log 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27701.log: /var/log/ceph/ceph-client.admin.66351.log 2024-01-12T01:08:19.996 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37425.log.gz 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27701.log.gz 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36437.log 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43626.log.gz 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66351.log.gz 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54581.log 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36437.log: gzip -5 --verbose -- 23.7%/var/log/ceph/ceph-client.admin.35765.log 2024-01-12T01:08:19.997 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36437.log.gz 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71139.log 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54581.log.gz 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52257.log 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35765.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.35765.log.gz 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45290.log 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71139.log.gz 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70515.log 2024-01-12T01:08:19.998 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52257.log.gz 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.45290.log: --verbose -- /var/log/ceph/ceph-client.admin.39676.log 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45290.log.gz 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51112.log 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70515.log.gz 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62367.log 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39676.log.gz 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53552.log 2024-01-12T01:08:19.999 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51112.log.gz 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36542.log 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62367.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.62367.log.gz 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.63733.log 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53552.log.gz 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70896.log 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36542.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36542.log.gz 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68645.log 2024-01-12T01:08:20.000 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63733.log.gz 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42698.log 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70896.log.gz 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.68645.log: --verbose -- /var/log/ceph/ceph-client.admin.59902.log 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68645.log.gz 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31099.log 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42698.log.gz 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28927.log 2024-01-12T01:08:20.001 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59902.log: /var/log/ceph/ceph-client.admin.31099.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.38864.log 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59902.log.gz 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31099.log.gz 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47515.log 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28927.log.gz 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29874.log 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38864.log.gz 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30419.log 2024-01-12T01:08:20.002 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47515.log.gz 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51654.log 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29874.log.gz 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39966.log 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54701.log 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30419.log.gz 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51654.log: gzip -5 --verbose -- 23.8%/var/log/ceph/ceph-client.admin.23912.log 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51654.log.gz 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39966.log.gz 2024-01-12T01:08:20.003 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62073.log 2024-01-12T01:08:20.004 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54701.log.gz 2024-01-12T01:08:20.004 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.23912.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26929.log 2024-01-12T01:08:20.004 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23912.log.gz 2024-01-12T01:08:20.004 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31082.log 2024-01-12T01:08:20.004 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62073.log.gz 2024-01-12T01:08:20.004 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61955.log 2024-01-12T01:08:20.004 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26929.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.26929.log.gz 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35457.log 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31082.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31082.log.gz 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40502.log 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61955.log.gz 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63270.log 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35457.log.gz 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27937.log 2024-01-12T01:08:20.005 INFO:teuthology.orchestra.run.smithi178.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.40502.log.gz 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52041.log 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63270.log: gzip 23.3%-5 -- replaced with /var/log/ceph/ceph-client.admin.63270.log.gz--verbose 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.32448.log 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27937.log.gz 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63022.log 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52041.log.gz 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61859.log 2024-01-12T01:08:20.006 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32448.log.gz 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28281.log 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr: 57.3% -- replaced with /var/log/ceph/ceph-client.admin.63022.log.gz 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55327.log 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61859.log.gz 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25420.log 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28281.log.gz 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60080.log 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55327.log.gz 2024-01-12T01:08:20.007 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40447.log 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25420.log.gz 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29594.log 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60080.log.gz 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57336.log 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40447.log.gz 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50159.log 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29594.log.gz 2024-01-12T01:08:20.008 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67489.log 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57336.log.gz 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54599.log 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50159.log.gz 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37808.log 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67489.log: /var/log/ceph/ceph-client.admin.54599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67489.log.gz 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.54599.log.gz 2024-01-12T01:08:20.009 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55499.log 2024-01-12T01:08:20.010 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37808.log.gz 2024-01-12T01:08:20.010 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.39739.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2024-01-12T01:08:20.010 INFO:teuthology.orchestra.run.smithi178.stderr: 17.2% -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2024-01-12T01:08:20.010 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23695.log 2024-01-12T01:08:20.010 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55499.log.gz 2024-01-12T01:08:20.010 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25895.log 2024-01-12T01:08:20.010 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28813.log 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23695.log.gz 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56798.log 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25895.log.gz 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26715.log 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55348.log 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.28813.log.gz 2024-01-12T01:08:20.011 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35217.log 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56798.log.gz 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26715.log.gz 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55348.log: /var/log/ceph/ceph-client.admin.41868.log 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55348.log.gz 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59001.log 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35217.log.gz 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35828.log 2024-01-12T01:08:20.012 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34452.log 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59001.log: /var/log/ceph/ceph-client.admin.35828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59001.log.gz 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42008.log 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.35828.log.gz 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62754.log 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34452.log.gz 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64013.log 2024-01-12T01:08:20.013 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42008.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.42008.log.gz 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41103.log 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62754.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.25629.log 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62754.log.gz 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64013.log: gzip /var/log/ceph/ceph-client.admin.41103.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57637.log 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64013.log.gz 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41103.log.gz 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42827.log 2024-01-12T01:08:20.014 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25629.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.25629.log.gz 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32808.log 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57637.log.gz 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26094.log 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42827.log.gz 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48031.log 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32808.log.gz 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45376.log 2024-01-12T01:08:20.015 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26094.log.gz 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48031.log.gz 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65496.log 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45376.log.gz 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66823.log 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gz 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49243.log 2024-01-12T01:08:20.016 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65496.log.gz 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55651.log 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66823.log.gz 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57422.log 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49243.log.gz 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30470.log 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55651.log.gz 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34197.log 2024-01-12T01:08:20.017 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57422.log.gz 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56820.log 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30470.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30470.log.gz 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64314.log 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34197.log.gz 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56820.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56820.log.gz-5 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-01-12T01:08:20.018 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64314.log.gz 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44316.log 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32028.log 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39655.log 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44316.log.gz 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32028.log.gz 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58307.log 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64894.log 2024-01-12T01:08:20.019 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39655.log.gz 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58307.log.gz 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.64894.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44925.log 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64894.log.gz 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46913.log 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45835.log 2024-01-12T01:08:20.020 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44925.log.gz 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26053.log 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46913.log.gz 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52127.log 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45835.log.gz 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30232.log 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26053.log.gz 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51032.log 2024-01-12T01:08:20.021 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52127.log.gz 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45595.log 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30232.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30232.log.gz 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44229.log 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51032.log.gz 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60545.log 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45595.log.gz 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28646.log 2024-01-12T01:08:20.022 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44229.log.gz 2024-01-12T01:08:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54111.log 2024-01-12T01:08:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48749.log 2024-01-12T01:08:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28646.log: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.60545.log.gz 2024-01-12T01:08:20.023 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28646.log.gz 2024-01-12T01:08:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29723.log 2024-01-12T01:08:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54111.log.gz 2024-01-12T01:08:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48369.log 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48749.log.gz 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.29723.log: -- /var/log/ceph/ceph-client.admin.32708.log 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29723.log.gz 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63905.log 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48369.log.gz 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35534.log 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32708.log.gz 2024-01-12T01:08:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34401.log 2024-01-12T01:08:20.025 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63905.log.gz 2024-01-12T01:08:20.025 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.35534.log: --verbose -- /var/log/ceph/ceph-client.admin.62496.log 2024-01-12T01:08:20.025 INFO:teuthology.orchestra.run.smithi178.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.35534.log.gz 2024-01-12T01:08:20.025 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64916.log 2024-01-12T01:08:20.025 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34401.log.gz 2024-01-12T01:08:20.025 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50581.log 2024-01-12T01:08:20.025 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62496.log.gz 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36227.log 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64916.log.gz 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39924.log 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50581.log.gz 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64271.log 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36227.log.gz 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.39924.log: --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr: 58.0% -- replaced with /var/log/ceph/ceph-client.admin.39924.log.gz 2024-01-12T01:08:20.026 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48856.log 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64271.log.gz 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25989.log 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36774.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50181.log 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48856.log.gz 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70190.log 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25989.log.gz 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31235.log 2024-01-12T01:08:20.027 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50181.log.gz 2024-01-12T01:08:20.028 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35098.log 2024-01-12T01:08:20.028 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70190.log.gz 2024-01-12T01:08:20.028 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53424.log /var/log/ceph/ceph-client.admin.31235.log: 2024-01-12T01:08:20.028 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31235.log.gz 2024-01-12T01:08:20.028 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66114.log 2024-01-12T01:08:20.028 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35098.log.gz 2024-01-12T01:08:20.028 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37341.log 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53424.log.gz 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53530.log 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66114.log.gz 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54742.log 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37341.log: gzip 23.7%-5 -- replaced with /var/log/ceph/ceph-client.admin.37341.log.gz--verbose 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.61658.log 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53530.log.gz 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29192.log 2024-01-12T01:08:20.029 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54742.log.gz 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34656.log 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61658.log.gz 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42720.log 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29192.log.gz 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44206.log 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34656.log.gz 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45615.log 2024-01-12T01:08:20.030 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42720.log.gz 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26306.log 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44206.log.gz 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37657.log 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45615.log.gz 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43454.log 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26306.log.gz 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31609.log 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37657.log.gz 2024-01-12T01:08:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.43454.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41690.log 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43454.log.gz 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30623.log 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31609.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31609.log.gz 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48347.log 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41690.log.gz 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43755.log 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30623.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30623.log.gz 2024-01-12T01:08:20.032 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60818.log 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48347.log.gz 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43755.log.gz 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64142.log 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60818.log.gz 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26392.log 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47364.log 2024-01-12T01:08:20.033 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64142.log.gz 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33993.log 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26392.log.gz 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66308.log 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47364.log.gz 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33048.log 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33993.log.gz 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49265.log 2024-01-12T01:08:20.034 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66308.log.gz 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56755.log 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33048.log.gz 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58231.log 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49265.log.gz 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67649.log 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56755.log.gz 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35807.log 2024-01-12T01:08:20.035 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58231.log.gz 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48224.log 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67649.log.gz 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23526.log 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35807.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.35807.log.gz 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68024.log 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48224.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.48224.log.gz 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36290.log 2024-01-12T01:08:20.036 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23526.log.gz 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68024.log.gz 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48598.log 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.36290.log.gz 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39487.log 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41062.log: 57.0% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34435.log 2024-01-12T01:08:20.037 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48598.log: 26.5%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.48598.log.gz--verbose 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.41789.log 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39487.log.gz 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43777.log 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34435.log.gz 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56300.log 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41789.log.gz 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24594.log 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43777.log.gz 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33268.log 2024-01-12T01:08:20.038 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56300.log.gz 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69080.log 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24594.log.gz 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27357.log 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33268.log.gz 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49579.log 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69080.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.69080.log.gz-5 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.64400.log 2024-01-12T01:08:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27357.log.gz 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55391.log 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49579.log.gz 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45269.log 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64400.log.gz 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47773.log 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55391.log.gzgzip 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.53638.log 2024-01-12T01:08:20.040 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45269.log.gz 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54299.log 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47773.log.gz 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70677.log 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53638.log.gz 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55541.log 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54299.log.gz 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31558.log 2024-01-12T01:08:20.041 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70677.log.gz 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68211.log 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55541.log.gz 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.31558.log: --verbose -- /var/log/ceph/ceph-client.admin.36942.log 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31558.log.gz 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61268.log 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68211.log.gz 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37320.log 2024-01-12T01:08:20.042 INFO:teuthology.orchestra.run.smithi178.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36942.log.gz 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66372.log 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61268.log.gz 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50398.log 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37320.log: 23.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51302.log 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37320.log.gz/var/log/ceph/ceph-client.admin.66372.log: 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66372.log.gz 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44882.log 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50398.log.gz 2024-01-12T01:08:20.043 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67632.log 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51302.log: 3.8% -- replaced with /var/log/ceph/ceph-client.admin.51302.log.gz 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60383.log 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44882.log.gz 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59271.log 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67632.log.gz 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44440.log 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60383.log.gz 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28238.log 2024-01-12T01:08:20.044 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59271.log.gz 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28345.log 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44440.log.gz 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28023.log 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28238.log.gz 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43691.log 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28345.log.gz 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66628.log 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28023.log.gz 2024-01-12T01:08:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43691.log.gz 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55069.log 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31828.log 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66628.log.gz 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27336.log 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55069.log.gz 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50968.log 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31828.log.gz 2024-01-12T01:08:20.046 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53467.log 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27336.log.gz 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59767.log 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50968.log.gz 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53467.log.gz 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60567.log 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59767.log.gz 2024-01-12T01:08:20.047 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42149.log 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66952.log 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60567.log.gz 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56507.log 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.42149.log.gz 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57293.log 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66952.log.gz 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62861.log 2024-01-12T01:08:20.048 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56507.log.gz 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64615.log 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57293.log.gz 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46019.log 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62861.log.gz 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64615.log: /var/log/ceph/ceph-client.admin.71058.log 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64615.log.gz 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23915.log 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46019.log.gz 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68076.log 2024-01-12T01:08:20.049 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71058.log.gz 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66737.log 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23915.log.gz 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42999.log 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68076.log.gz 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29637.log 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66737.log.gz 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30946.log 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42999.log.gz 2024-01-12T01:08:20.050 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57508.log 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29637.log.gz 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39251.log 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30946.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30946.log.gz 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57508.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.57508.log.gz--verbose 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.28582.log 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63819.log 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27089.log 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39251.log.gz 2024-01-12T01:08:20.051 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28582.log.gz 2024-01-12T01:08:20.052 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.63819.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52687.log 2024-01-12T01:08:20.052 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63819.log.gz 2024-01-12T01:08:20.052 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51420.log 2024-01-12T01:08:20.052 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27089.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.27089.log.gz 2024-01-12T01:08:20.052 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31626.log 2024-01-12T01:08:20.052 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52687.log.gz 2024-01-12T01:08:20.052 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27072.log 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51420.log.gz 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62560.log 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31626.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31626.log.gz 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60262.log 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27072.log.gz 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68864.log 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62582.log 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62560.log.gz 2024-01-12T01:08:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60262.log.gz 2024-01-12T01:08:20.054 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39401.log 2024-01-12T01:08:20.054 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68864.log.gz 2024-01-12T01:08:20.054 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41571.log 2024-01-12T01:08:20.054 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-01-12T01:08:20.054 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62582.log.gz 2024-01-12T01:08:20.054 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39401.log.gz 2024-01-12T01:08:20.054 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41571.log: 52.8% -- replaced with /var/log/ceph/ceph-client.admin.41571.log.gz 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31133.log 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38821.log 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31167.log 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31133.log.gz 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30878.log 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38821.log.gz 2024-01-12T01:08:20.055 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26177.log 2024-01-12T01:08:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31167.log: /var/log/ceph/ceph-client.admin.30878.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31167.log.gz 2024-01-12T01:08:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54154.log 2024-01-12T01:08:20.056 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30878.log.gz 2024-01-12T01:08:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2024-01-12T01:08:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26177.log: /var/log/ceph/ceph-client.admin.54154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26177.log.gz 2024-01-12T01:08:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47988.log 2024-01-12T01:08:20.056 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54154.log.gz 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36458.log 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47988.log.gz 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36458.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.36458.log.gz 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50052.log 2024-01-12T01:08:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38503.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47170.log 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64120.log 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50052.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50052.log.gz-5 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.40545.log 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47170.log.gz 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41269.log 2024-01-12T01:08:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64120.log.gz 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43884.log 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40545.log.gz 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56384.log 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41269.log.gz 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34843.log 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43884.log.gz 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25374.log 2024-01-12T01:08:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56384.log.gz 2024-01-12T01:08:20.060 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59829.log 2024-01-12T01:08:20.060 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34843.log.gz 2024-01-12T01:08:20.060 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63149.log 2024-01-12T01:08:20.060 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25374.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.25374.log.gz 2024-01-12T01:08:20.060 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20019.log 2024-01-12T01:08:20.060 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59829.log.gz 2024-01-12T01:08:20.060 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45758.log 2024-01-12T01:08:20.063 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63149.log.gz 2024-01-12T01:08:20.063 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28560.log 2024-01-12T01:08:20.063 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.20019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20019.log.gz 2024-01-12T01:08:20.063 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70542.log 2024-01-12T01:08:20.063 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45758.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53573.log 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45758.log.gz/var/log/ceph/ceph-client.admin.28560.log: 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28560.log.gz 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49439.log 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70542.log.gz 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66652.log 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53573.log.gz 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26073.log 2024-01-12T01:08:20.064 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49439.log.gz 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66652.log: 28.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.66652.log.gz-5 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.43820.log 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40024.log 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26073.log.gz 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43820.log.gz 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24680.log 2024-01-12T01:08:20.065 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46806.log 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr: 22.6% -- replaced with /var/log/ceph/ceph-client.admin.40024.log.gz 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50009.log 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24680.log.gz 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46806.log.gz 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54481.log 2024-01-12T01:08:20.066 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50009.log.gz 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41831.log 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44860.log 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54481.log.gz 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63190.log 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41831.log.gz 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36627.log 2024-01-12T01:08:20.067 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44860.log.gz 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28496.log 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63190.log.gz 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44817.log 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66713.log 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36627.log.gz/var/log/ceph/ceph-client.admin.28496.log: 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28496.log.gz 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44817.log.gz 2024-01-12T01:08:20.068 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64464.log 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60001.log 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66713.log: 52.0% -- replaced with /var/log/ceph/ceph-client.admin.66713.log.gz 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30453.log 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64464.log.gz 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66608.log 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60001.log.gz 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30453.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30453.log.gz 2024-01-12T01:08:20.069 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64572.log 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.66608.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34690.log 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66608.log.gz 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.64572.log: --verbose -- /var/log/ceph/ceph-client.admin.53702.log 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64572.log.gz 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32508.log 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34690.log.gz 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60640.log 2024-01-12T01:08:20.070 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53702.log: 0.0%/var/log/ceph/ceph-client.admin.32508.log: -- replaced with /var/log/ceph/ceph-client.admin.53702.log.gz 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46998.log 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32508.log.gz 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37851.log 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60640.log.gz 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24256.log 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.46998.log.gz 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51197.log 2024-01-12T01:08:20.071 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37851.log.gz 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67382.log 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24256.log.gz 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48878.log 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51197.log.gz 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62132.log 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67382.log.gz 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34673.log 2024-01-12T01:08:20.072 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48878.log.gz 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54068.log 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62132.log.gz 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56202.log 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34673.log.gz 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30997.log 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54068.log.gz 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.56202.log: -- /var/log/ceph/ceph-client.admin.30249.log 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56202.log.gz 2024-01-12T01:08:20.073 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37722.log 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30997.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30997.log.gz 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67124.log 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30249.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30249.log.gz 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60465.log 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37722.log.gz 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60282.log 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67124.log.gz 2024-01-12T01:08:20.074 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57938.log 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60465.log: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.60465.log.gz 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33208.log 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60282.log.gz 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53215.log 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57938.log.gz 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.33208.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.66866.log 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33208.log.gz 2024-01-12T01:08:20.075 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25687.log 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53215.log.gz 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64228.log 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66866.log.gz 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29937.log 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25687.log.gz 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47321.log 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64228.log.gz 2024-01-12T01:08:20.076 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53659.log 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29937.log.gz 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65657.log 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47321.log.gz 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31988.log 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53659.log.gz 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33976.log 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65657.log.gz 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60322.log 2024-01-12T01:08:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31988.log.gz 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33976.log.gz 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32848.log 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43927.log 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%/var/log/ceph/ceph-client.admin.40751.log: -- replaced with /var/log/ceph/ceph-client.admin.60322.log.gz 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60404.log 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz/var/log/ceph/ceph-client.admin.32848.log: 2024-01-12T01:08:20.078 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32848.log.gz 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59292.log 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43927.log.gz 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31337.log 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60404.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.42529.log 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60404.log.gz 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59292.log.gz 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38759.log 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31337.log: /var/log/ceph/ceph-client.admin.42529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52601.log 2024-01-12T01:08:20.079 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% 23.7% -- replaced with /var/log/ceph/ceph-client.admin.42529.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31337.log.gz 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44056.log 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68265.log 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38759.log.gz 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52601.log.gz 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67403.log 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44056.log.gz 2024-01-12T01:08:20.080 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37743.log 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68265.log.gz 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53831.log 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67403.log.gz 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32928.log 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37743.log.gz 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62195.log 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2024-01-12T01:08:20.081 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32928.log.gz 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40687.log 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62195.log.gz 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62711.log 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41909.log 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40687.log.gz 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50073.log 2024-01-12T01:08:20.082 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62711.log.gz 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38109.log 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41909.log.gz 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48964.log 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50073.log.gz 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38109.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38109.log.gz-5 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.29444.log 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49157.log 2024-01-12T01:08:20.083 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48964.log.gz 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32888.log 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29444.log.gz 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46289.log 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49157.log.gz 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23752.log 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32888.log.gz 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54339.log 2024-01-12T01:08:20.084 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46289.log.gz 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28066.log 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23752.log.gz 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61072.log 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54339.log.gz 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62324.log 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28066.log.gz 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59041.log 2024-01-12T01:08:20.085 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61072.log.gz 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51483.log 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62324.log.gz 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62281.log 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59041.log: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.59041.log.gz 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69460.log 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51483.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.51483.log.gz 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68293.log 2024-01-12T01:08:20.086 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54832.log 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62281.log.gz 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30062.log 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69460.log.gz 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68293.log.gz 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28732.log 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54832.log.gz 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41352.log 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46477.log 2024-01-12T01:08:20.087 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30062.log.gz 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28732.log.gz 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44526.log 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41352.log.gz 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40587.log 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46477.log: 56.8% -- replaced with /var/log/ceph/ceph-client.admin.46477.log.gz 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59547.log 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44526.log.gz 2024-01-12T01:08:20.088 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34605.log 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40587.log.gz 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52300.log 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59547.log.gz 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34877.log 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34605.log.gz 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33288.log 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52300.log.gz 2024-01-12T01:08:20.089 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25933.log 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34877.log.gz 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50310.log 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33288.log.gz 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45675.log 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25933.log.gz 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61328.log 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50310.log.gz 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60193.log 2024-01-12T01:08:20.090 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45675.log.gz 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34282.log 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61328.log.gz 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56027.log 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60193.log.gz 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57680.log 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34282.log.gz 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55434.log 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56027.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70325.log 2024-01-12T01:08:20.091 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56027.log.gz 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57680.log.gz 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59942.log 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55434.log.gz 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50906.log 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70325.log.gz 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70623.log 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59942.log: 56.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62302.log 2024-01-12T01:08:20.092 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50906.log: -- replaced with /var/log/ceph/ceph-client.admin.59942.log.gz 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55809.log 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50906.log.gz 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70623.log.gz 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29980.log 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62302.log.gz 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49538.log 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55809.log.gz 2024-01-12T01:08:20.093 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58772.log 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29980.log.gz 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48160.log 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49538.log: 55.1% -- replaced with /var/log/ceph/ceph-client.admin.49538.log.gz 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35996.log 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58772.log.gz 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69865.log 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48160.log.gz 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.35996.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29465.log 2024-01-12T01:08:20.094 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.35996.log.gz 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46249.log 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69865.log.gz 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26521.log 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29465.log.gz 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30657.log 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46249.log.gz 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43042.log 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26521.log.gz 2024-01-12T01:08:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70812.log 2024-01-12T01:08:20.096 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30657.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30657.log.gz 2024-01-12T01:08:20.096 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66802.log 2024-01-12T01:08:20.096 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43042.log.gz 2024-01-12T01:08:20.096 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59564.log 2024-01-12T01:08:20.096 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37572.log 2024-01-12T01:08:20.096 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70812.log.gz 2024-01-12T01:08:20.096 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66802.log.gz 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54003.log 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59564.log.gz 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63862.log 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37572.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.37572.log.gz 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24960.log 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54003.log.gz 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52967.log 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63862.log.gz 2024-01-12T01:08:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39759.log 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24960.log.gz 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50701.log 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52967.log.gz 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34707.log 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39759.log: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.39759.log.gz 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57357.log 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50701.log.gz 2024-01-12T01:08:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27106.log 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34707.log.gz 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28410.log 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57357.log.gz 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33408.log 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27106.log.gz 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58131.log 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28410.log.gz 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60136.log 2024-01-12T01:08:20.099 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33408.log.gz 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59788.log 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58131.log.gz 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50378.log 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60136.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.55047.log 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60136.log.gz 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59788.log.gz 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.50378.log: -- /var/log/ceph/ceph-client.admin.58634.log 2024-01-12T01:08:20.100 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50378.log.gz 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31031.log 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55047.log.gz 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28431.log 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58634.log.gz 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51611.log 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31031.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31031.log.gz 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28431.log.gz 2024-01-12T01:08:20.101 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36606.log 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.51611.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31660.log 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.51611.log.gz 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63352.log 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36606.log: 23.9%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.36606.log.gz--verbose 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.54918.log 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31660.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31660.log.gz 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34826.log 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63352.log.gz 2024-01-12T01:08:20.102 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34044.log 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54918.log.gz 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33648.log 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34826.log.gz 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25132.log 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34044.log.gz 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34299.log 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33648.log.gz 2024-01-12T01:08:20.103 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26135.log 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25132.log.gz 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34928.log 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34299.log.gz 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37278.log 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26135.log.gz 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32968.log 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34928.log.gz 2024-01-12T01:08:20.104 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71085.log 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37278.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.37278.log.gz 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42029.log 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32968.log.gz 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67231.log 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71085.log.gz 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27293.log 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42029.log.gz 2024-01-12T01:08:20.105 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27004.log 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67231.log.gz 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50823.log 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27293.log.gz 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38543.log 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27004.log.gz 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71383.log 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50823.log.gz 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45855.log 2024-01-12T01:08:20.106 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38543.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.38543.log.gz 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55241.log 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71383.log.gz 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46329.log 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45855.log.gz 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48182.log 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55241.log.gz 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61247.log 2024-01-12T01:08:20.107 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46329.log.gz 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33840.log 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48182.log.gz 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.61247.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67596.log 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61247.log.gz 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60425.log 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33840.log.gz 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47407.log 2024-01-12T01:08:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67596.log: /var/log/ceph/ceph-client.admin.60425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67596.log.gz 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61092.log 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60425.log.gz 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35285.log 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47407.log.gz 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51912.log 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61092.log.gz 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29315.log 2024-01-12T01:08:20.109 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35285.log.gz 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56064.log 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51912.log.gz 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41472.log 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29315.log.gz 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34350.log 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47472.log 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56064.log.gz 2024-01-12T01:08:20.110 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41472.log.gz 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50561.log 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34350.log.gz 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63063.log 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47472.log.gz 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23835.log 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50561.log.gz 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45419.log 2024-01-12T01:08:20.111 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63063.log.gz 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.23835.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34180.log 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23835.log.gz 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30572.log 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45419.log.gz 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20144.log 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38842.log 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34180.log.gz 2024-01-12T01:08:20.112 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30572.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30572.log.gz 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.20144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20144.log.gz 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26946.log 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38842.log.gz 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37829.log 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41653.log: /var/log/ceph/ceph-client.admin.26946.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56550.log 2024-01-12T01:08:20.113 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26946.log.gz 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.37829.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67081.log 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37829.log.gz 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34741.log 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56550.log.gz 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52816.log 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67081.log.gz 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41019.log 2024-01-12T01:08:20.114 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53595.log 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34741.log.gz 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52816.log: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.52816.log.gz 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35132.log 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41019.log.gz 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54897.log 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53595.log.gz 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28173.log 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35132.log.gz 2024-01-12T01:08:20.115 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.54897.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34775.log 2024-01-12T01:08:20.116 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54897.log.gz 2024-01-12T01:08:20.116 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51156.log 2024-01-12T01:08:20.116 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28173.log.gz 2024-01-12T01:08:20.116 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30895.log 2024-01-12T01:08:20.116 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34775.log.gz 2024-01-12T01:08:20.116 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2024-01-12T01:08:20.116 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51156.log.gz 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70433.log 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30895.log.gz 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44662.log 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44387.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42170.log 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70433.log.gz 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29745.log 2024-01-12T01:08:20.117 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44662.log.gz 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61425.log 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42170.log.gz 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19967.log 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29745.log.gz 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61425.log.gz 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.19967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44509.log 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19967.log.gz 2024-01-12T01:08:20.118 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26435.log 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41372.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32228.log 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44509.log.gz 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38581.log 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63540.log 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26435.log.gz 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32228.log: 28.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.32228.log.gz 2024-01-12T01:08:20.119 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.36080.log 2024-01-12T01:08:20.120 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38581.log.gz 2024-01-12T01:08:20.120 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55558.log 2024-01-12T01:08:20.120 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63540.log.gz 2024-01-12T01:08:20.120 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42391.log 2024-01-12T01:08:20.120 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36080.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.36080.log.gz 2024-01-12T01:08:20.120 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25515.log 2024-01-12T01:08:20.120 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55558.log.gz 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55133.log 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42391.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.42391.log.gz 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56712.log 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25515.log.gz 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26801.log 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55133.log.gz 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30793.log 2024-01-12T01:08:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56712.log.gz 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.26801.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65637.log 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26801.log.gz 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55370.log 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30793.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30793.log.gz 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51379.log 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65637.log.gz 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65270.log 2024-01-12T01:08:20.122 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55370.log.gz 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56669.log 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51379.log.gz 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65270.log: 23.0%/var/log/ceph/ceph-client.admin.56669.log: -- replaced with /var/log/ceph/ceph-client.admin.65270.log.gz 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56220.log 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56669.log.gz 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68184.log 2024-01-12T01:08:20.123 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60835.log 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56220.log.gz 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67532.log 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68184.log.gz 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44351.log 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60835.log.gz 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35251.log 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67532.log.gz 2024-01-12T01:08:20.124 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34554.log 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44351.log.gz 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62962.log 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35251.log.gz 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69351.log 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34554.log.gz 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29616.log 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62962.log.gz 2024-01-12T01:08:20.125 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56841.log 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69351.log.gz 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32108.log 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29616.log.gz 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41930.log 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56841.log.gz 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23722.log 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32108.log.gz 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60176.log 2024-01-12T01:08:20.126 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41930.log.gz 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45161.log 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50245.log 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60176.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60176.log.gz 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.23722.log.gz 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70839.log 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45161.log.gz 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2024-01-12T01:08:20.127 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50245.log.gz 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59624.log 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70839.log.gz 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42677.log 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28216.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38195.log 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59624.log.gz 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43107.log 2024-01-12T01:08:20.128 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42677.log.gz 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69270.log 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38195.log.gz 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43107.log.gz 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64722.log 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31218.log 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69270.log.gz/var/log/ceph/ceph-client.admin.64722.log: 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64722.log.gz 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52730.log 2024-01-12T01:08:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27615.log 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31218.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31218.log.gz 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29916.log 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52730.log.gz 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46455.log 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27615.log.gz 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25356.log 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29916.log.gz 2024-01-12T01:08:20.130 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29895.log 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46455.log.gz 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44280.log 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25356.log.gz 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29895.log.gz 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62904.log 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44280.log.gz 2024-01-12T01:08:20.131 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34061.log 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51319.log 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62904.log.gz 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44740.log 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34061.log.gz 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33188.log 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51319.log.gz 2024-01-12T01:08:20.132 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58383.log 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44740.log.gz 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32608.log /var/log/ceph/ceph-client.admin.33188.log: 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33188.log.gz 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31065.log 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58383.log: 26.5% -- replaced with /var/log/ceph/ceph-client.admin.58383.log.gz 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57142.log 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32608.log.gz 2024-01-12T01:08:20.133 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44579.log 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31065.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31065.log.gz 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69784.log 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57142.log.gz 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46498.log 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44579.log.gz 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23641.log 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69784.log.gz 2024-01-12T01:08:20.134 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34622.log 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46498.log.gz 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63478.log 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23641.log.gz 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54640.log 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34622.log.gz 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25801.log 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63478.log.gz 2024-01-12T01:08:20.135 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59644.log 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54640.log.gz 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29788.log 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25801.log.gz 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52709.log 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59644.log.gz 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45959.log 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29788.log.gz 2024-01-12T01:08:20.136 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26887.log 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52709.log.gz 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32748.log 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45959.log.gz 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60855.log 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26887.log.gz 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27126.log 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32748.log.gz 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59866.log 2024-01-12T01:08:20.137 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62646.log 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60855.log.gz 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27126.log.gz 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66222.log 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59866.log.gz 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65247.log 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62646.log.gz 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33857.log 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66222.log.gz 2024-01-12T01:08:20.138 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65856.log 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65247.log.gz 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56045.log 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33857.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33908.log 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65856.log.gz 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24745.log 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56045.log.gz 2024-01-12T01:08:20.139 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31320.log 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33908.log.gz 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62216.log 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24745.log.gz 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48792.log 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31320.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31320.log.gz 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33942.log 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62216.log.gz 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39718.log 2024-01-12T01:08:20.140 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48792.log.gz 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49518.log 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33942.log.gz 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35081.log 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39718.log.gz 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58597.log 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49518.log: 55.1% -- replaced with /var/log/ceph/ceph-client.admin.49518.log.gz 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65755.log 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35081.log.gz 2024-01-12T01:08:20.141 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60099.log 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58597.log.gz 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55948.log 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65755.log: 56.2% -- replaced with /var/log/ceph/ceph-client.admin.65755.log.gz 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67038.log 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60099.log.gz 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32248.log 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55948.log.gz 2024-01-12T01:08:20.142 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31048.log 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67038.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.67038.log.gz-- 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33708.log 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32248.log.gz 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37068.log 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31048.log.gz 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37110.log 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33708.log.gz 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56691.log 2024-01-12T01:08:20.143 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37068.log: /var/log/ceph/ceph-client.admin.37110.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.37068.log.gz 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63583.log 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37110.log.gz 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.56691.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45097.log 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56691.log.gz 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44641.log 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63583.log.gz 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25153.log 2024-01-12T01:08:20.144 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45097.log.gz 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36669.log 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44641.log.gz 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53090.log 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25153.log.gz 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58442.log 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36669.log.gz 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68130.log 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53090.log.gz 2024-01-12T01:08:20.145 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45398.log 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58442.log.gz 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28830.log 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68130.log.gz 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53256.log 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45398.log.gz 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65393.log 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28830.log.gz 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36185.log 2024-01-12T01:08:20.146 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53256.log.gz 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34809.log 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65393.log.gz 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60932.log 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36185.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.36185.log.gz 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38447.log 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34809.log: 26.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42634.log 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34809.log.gz 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60932.log.gz 2024-01-12T01:08:20.147 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38447.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38447.log.gz-5 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:--verbose -- /var/log/ceph/ceph-client.admin.40405.log 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28885.log 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42634.log.gz 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48053.log 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28885.log.gz 2024-01-12T01:08:20.148 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34146.log 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44405.log 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48053.log.gz 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34146.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50638.log 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34146.log.gz 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44405.log.gz 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36816.log 2024-01-12T01:08:20.149 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52623.log 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50638.log.gz 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60912.log 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.36816.log.gz 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52623.log.gz 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25416.log 2024-01-12T01:08:20.150 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60912.log.gz 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56970.log 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42129.log 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25416.log.gz 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56970.log.gz 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35115.log 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.42129.log.gz 2024-01-12T01:08:20.151 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36143.log 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60019.log 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35115.log.gz 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41851.log 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36143.log: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.36143.log.gz 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30130.log 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60019.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60019.log.gz 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.39801.log 2024-01-12T01:08:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41851.log: /var/log/ceph/ceph-client.admin.30130.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.41851.log.gz 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30130.log.gz 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54025.log 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33448.log 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39801.log: gzip -5 --verbose 56.2% -- replaced with /var/log/ceph/ceph-client.admin.39801.log.gz 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.54025.log: /var/log/ceph/ceph-client.admin.58689.log 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54025.log.gz 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56992.log 2024-01-12T01:08:20.153 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33448.log.gz 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58689.log.gz 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43257.log 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64249.log 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56992.log.gz 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.43257.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47881.log 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43257.log.gz 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.64249.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47601.log 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64249.log.gz 2024-01-12T01:08:20.154 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62925.log 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47881.log.gz 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63499.log 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47601.log.gz 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.62925.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51052.log 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62925.log.gz 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32728.log 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63499.log.gz 2024-01-12T01:08:20.155 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58614.log 2024-01-12T01:08:20.156 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51052.log.gz 2024-01-12T01:08:20.156 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25412.log 2024-01-12T01:08:20.156 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32728.log.gz 2024-01-12T01:08:20.156 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67468.log 2024-01-12T01:08:20.156 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58614.log.gz 2024-01-12T01:08:20.156 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43407.log 2024-01-12T01:08:20.156 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25412.log.gz 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64421.log 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67468.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.67468.log.gz--verbose 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:-- /var/log/ceph/ceph-client.admin.58265.log 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43407.log: /var/log/ceph/ceph-client.admin.64421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43407.log.gz 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46185.log 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64421.log.gz 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37467.log 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58265.log.gz 2024-01-12T01:08:20.157 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39697.log 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46185.log.gz 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61994.log 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37467.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37467.log.gz 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49686.log 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39697.log.gz 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40107.log 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61994.log.gz 2024-01-12T01:08:20.158 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55262.log 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49686.log.gz 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63561.log 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40107.log.gz 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62625.log 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55262.log.gz 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27314.log 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63561.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.63561.log.gz 2024-01-12T01:08:20.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59604.log 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62625.log.gz 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32788.log 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27314.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51977.log 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59604.log.gz 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56949.log 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32788.log.gz 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47494.log 2024-01-12T01:08:20.160 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27164.log 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51977.log.gz 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56949.log.gz 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47494.log.gz 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27164.log.gz 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69892.log 2024-01-12T01:08:20.161 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69107.log 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67167.log 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69892.log.gz 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69107.log.gz 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27915.log 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67167.log.gz 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.16287.log 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69433.log 2024-01-12T01:08:20.162 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27915.log.gz/var/log/ceph/ceph.tmp-client.admin.16287.log: 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68564.log 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.16287.log.gz 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71031.log 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69433.log.gz 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68564.log.gz 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39358.log 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30023.log 2024-01-12T01:08:20.163 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71031.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36563.log 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49399.log 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr: 28.3% -- replaced with /var/log/ceph/ceph-client.admin.30023.log.gz 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61211.log 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36563.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.36563.log.gz 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49399.log.gz 2024-01-12T01:08:20.164 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62111.log 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71410.log 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61211.log.gz 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58710.log 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62111.log.gz 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27508.log 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.71410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71410.log.gz 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58710.log.gz 2024-01-12T01:08:20.165 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43128.log 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27508.log.gz 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50138.log 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz/var/log/ceph/ceph-client.admin.43128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55788.log 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43128.log.gz 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50138.log.gz 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34248.log 2024-01-12T01:08:20.166 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25573.log 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55788.log.gz 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65434.log 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34248.log.gz 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25573.log.gz 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25591.log 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65434.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.65434.log.gz 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23614.log 2024-01-12T01:08:20.167 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25591.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.25591.log.gz 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42570.log 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23614.log.gz 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43497.log 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42570.log.gz 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36353.log 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27184.log 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43497.log.gz 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38216.log 2024-01-12T01:08:20.168 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40627.log 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27184.log: 0.0% 23.3% -- replaced with /var/log/ceph/ceph-client.admin.27184.log.gz 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36353.log.gz 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38216.log.gz 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33548.log 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29056.log 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40627.log: 0.0%/var/log/ceph/ceph-client.admin.33548.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33548.log.gz-- 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29551.log 2024-01-12T01:08:20.169 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26414.log 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29056.log.gz 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40627.log.gz 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61779.log 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29551.log.gz 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58942.log 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26414.log.gz 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61779.log.gz 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61346.log 2024-01-12T01:08:20.170 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55711.log 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58942.log.gz 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61346.log.gz 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36101.log 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55711.log.gz 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45635.log 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54940.log 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36101.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62818.log 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45635.log: 2024-01-12T01:08:20.171 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45635.log.gz 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27829.log 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54940.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.54940.log.gz-5 --verbose -- /var/log/ceph/ceph-client.admin.65224.log 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62818.log.gz 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32948.log 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27829.log.gz 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49643.log 2024-01-12T01:08:20.172 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64959.log 2024-01-12T01:08:20.173 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65224.log.gz/var/log/ceph/ceph-client.admin.32948.log: 2024-01-12T01:08:20.173 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32948.log.gz 2024-01-12T01:08:20.173 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62259.log 2024-01-12T01:08:20.173 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49643.log.gz 2024-01-12T01:08:20.173 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64959.log.gz 2024-01-12T01:08:20.290 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50802.log 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62259.log.gz 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58002.log 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69514.log 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50802.log.gz/var/log/ceph/ceph-client.admin.58002.log: 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56364.log 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58002.log.gz 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36332.log 2024-01-12T01:08:20.291 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69514.log.gz 2024-01-12T01:08:20.292 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56364.log.gz 2024-01-12T01:08:20.292 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31745.log 2024-01-12T01:08:20.292 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32988.log 2024-01-12T01:08:20.292 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36332.log: /var/log/ceph/ceph-client.admin.31745.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.36332.log.gz 2024-01-12T01:08:20.292 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31745.log.gz 2024-01-12T01:08:20.292 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28152.log 2024-01-12T01:08:20.292 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32988.log.gz 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65291.log 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35576.log 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28152.log.gz 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50865.log 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65291.log.gz 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62431.log 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28906.log 2024-01-12T01:08:20.293 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50865.log.gz 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.35576.log.gz 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62431.log.gz 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24702.log 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.28906.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30266.log 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28906.log.gz 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64357.log 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24702.log.gz/var/log/ceph/ceph-client.admin.30266.log: 2024-01-12T01:08:20.294 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69378.log 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30266.log.gz 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64357.log.gz 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35510.log 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41205.log 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69378.log.gz 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44423.log 2024-01-12T01:08:20.295 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35510.log: 53.4% -- replaced with /var/log/ceph/ceph-client.admin.35510.log.gz 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52859.log 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41205.log.gz 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45695.log 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44423.log.gz 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24019.log 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52859.log.gz 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56884.log 2024-01-12T01:08:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45695.log.gz 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34486.log 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24019.log.gz 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34503.log 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56884.log.gz 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36879.log 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34486.log.gz 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64787.log 2024-01-12T01:08:20.297 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34503.log.gz 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46935.log 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36879.log: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36879.log.gz 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68699.log 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64787.log.gz 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66136.log 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46935.log.gz 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66566.log 2024-01-12T01:08:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68699.log.gz 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43841.log 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66136.log.gz 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27743.log 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66566.log.gz 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32268.log 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43841.log.gz 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65942.log 2024-01-12T01:08:20.299 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27743.log.gz 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.32268.log: -- /var/log/ceph/ceph-client.admin.69946.log 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32268.log.gz 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33508.log 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65942.log.gz 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48706.log 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69946.log.gz 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-01-12T01:08:20.300 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33508.log.gz 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62840.log 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48706.log.gz 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39904.log 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40066.log: 18.5% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60758.log /var/log/ceph/ceph-client.admin.62840.log: 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62840.log.gz 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65476.log 2024-01-12T01:08:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39904.log: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.39904.log.gz 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67296.log 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60758.log.gz 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37089.log 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65476.log.gz 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45484.log 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67296.log.gz 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33789.log 2024-01-12T01:08:20.302 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37089.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.37089.log.gz 2024-01-12T01:08:20.303 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25913.log 2024-01-12T01:08:20.303 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45484.log.gz 2024-01-12T01:08:20.303 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33628.log 2024-01-12T01:08:20.303 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33789.log: /var/log/ceph/ceph-client.admin.25913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57551.log 2024-01-12T01:08:20.303 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33789.log.gz 2024-01-12T01:08:20.303 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25913.log.gz 2024-01-12T01:08:20.303 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45655.log 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33628.log.gz 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37872.log 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57551.log.gz 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70244.log 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63413.log 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45655.log.gz 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37872.log.gz 2024-01-12T01:08:20.304 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33308.log 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70244.log.gz 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54620.log 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63413.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63413.log.gz 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.30861.log 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33308.log.gz 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68810.log 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54620.log.gz 2024-01-12T01:08:20.305 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36038.log 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30861.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30861.log.gz 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60952.log 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68810.log.gz 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36038.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36038.log.gz 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58478.log 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60952.log.gz 2024-01-12T01:08:20.306 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35954.log 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67997.log 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58478.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.58478.log.gz 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.29140.log 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35954.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.35954.log.gz 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47537.log 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67997.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58828.log 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67997.log.gz 2024-01-12T01:08:20.307 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29140.log.gz 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24938.log 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47537.log.gz 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61718.log 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58828.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31201.log 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24938.log: -- replaced with /var/log/ceph/ceph-client.admin.58828.log.gz 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24938.log.gz 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60604.log 2024-01-12T01:08:20.308 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61718.log.gz 2024-01-12T01:08:20.309 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58578.log 2024-01-12T01:08:20.309 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31201.log: 1.2%/var/log/ceph/ceph-client.admin.60604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34945.log 2024-01-12T01:08:20.309 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31201.log.gz 2024-01-12T01:08:20.309 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60604.log.gz 2024-01-12T01:08:20.309 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37026.log 2024-01-12T01:08:20.309 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58578.log.gz 2024-01-12T01:08:20.309 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52106.log 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34945.log.gz 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37026.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37026.log.gz 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45876.log 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49329.log 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52106.log.gz 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28539.log 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45876.log.gz 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34962.log 2024-01-12T01:08:20.310 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49329.log.gz 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29401.log 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28539.log.gz 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67059.log 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34962.log.gz 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35491.log 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29401.log.gz 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.67059.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64378.log 2024-01-12T01:08:20.311 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67059.log.gz 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41451.log 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35491.log.gz 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43433.log 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64378.log.gz 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.41451.log.gz 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55892.log 2024-01-12T01:08:20.312 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43433.log.gz 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30725.log /var/log/ceph/ceph-client.admin.38657.log: 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65002.log 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55892.log.gz 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.30725.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53939.log 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30725.log.gz 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56141.log 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65002.log.gz/var/log/ceph/ceph-client.admin.53939.log: 2024-01-12T01:08:20.313 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50095.log 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53939.log.gz 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56141.log.gz 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43343.log 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31286.log 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50095.log.gz 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53341.log 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43343.log.gz 2024-01-12T01:08:20.314 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50742.log 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31286.log.gz 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43906.log 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53341.log.gz 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47644.log 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50742.log.gz 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58421.log 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43906.log.gz 2024-01-12T01:08:20.315 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59487.log 2024-01-12T01:08:20.316 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47644.log.gz 2024-01-12T01:08:20.316 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69919.log 2024-01-12T01:08:20.316 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58421.log.gz 2024-01-12T01:08:20.316 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-01-12T01:08:20.316 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59487.log.gz 2024-01-12T01:08:20.316 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44600.log 2024-01-12T01:08:20.316 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69919.log.gz 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35013.log 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46700.log 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44600.log.gz 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35013.log.gz 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48663.log 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46700.log.gz 2024-01-12T01:08:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65164.log 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36500.log 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48663.log.gz 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70569.log 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65164.log.gz 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26349.log 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.36500.log.gz 2024-01-12T01:08:20.318 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65921.log 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70569.log.gz 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.26349.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53874.log 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26349.log.gz 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23838.log 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65921.log.gz 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66007.log 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53874.log.gz 2024-01-12T01:08:20.319 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57121.log 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23838.log.gz 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26500.log 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.66007.log.gz 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35933.log 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57121.log.gz 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53361.log 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26500.log.gz 2024-01-12T01:08:20.320 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26543.log 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.35933.log.gz 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48283.log 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53361.log.gz 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.26543.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25839.log 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26543.log.gz 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69622.log 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48283.log.gz 2024-01-12T01:08:20.321 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48899.log 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25839.log.gz 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57099.log 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.69622.log.gz 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46393.log 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48899.log.gz 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28130.log 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57099.log.gz 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46393.log.gz 2024-01-12T01:08:20.322 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54132.log 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50224.log 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28130.log.gz 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25318.log 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54132.log.gz 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44121.log 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50224.log.gz 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40977.log 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52924.log 2024-01-12T01:08:20.323 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25318.log.gz 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44121.log.gz 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46892.log 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40977.log.gz 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44544.log 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52924.log: /var/log/ceph/ceph-client.admin.46892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56587.log 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52924.log.gz 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46892.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62410.log 2024-01-12T01:08:20.324 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44544.log.gz 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64486.log 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56587.log.gz 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58326.log 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62410.log.gz 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64486.log.gz 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31808.log 2024-01-12T01:08:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59746.log 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58326.log.gz 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53048.log 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31808.log.gz 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34129.log 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59746.log.gz 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52558.log 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51462.log 2024-01-12T01:08:20.326 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53048.log.gz 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34129.log.gz 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52558.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32168.log 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52558.log.gz 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25280.log 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51462.log.gz 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26220.log 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32168.log.gz 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58497.log 2024-01-12T01:08:20.327 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25280.log.gz 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38001.log 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26220.log.gz 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35302.log 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58497.log.gz 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54559.log 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38001.log.gz 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2024-01-12T01:08:20.328 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35302.log.gz 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42429.log 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54559.log.gz 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.39380.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26758.log 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30810.log 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.42429.log.gz 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31643.log 2024-01-12T01:08:20.329 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26758.log.gz 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.30810.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40647.log 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30810.log.gz 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31643.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31643.log.gz 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53896.log 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46616.log 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40647.log.gz 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37383.log 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53896.log.gz 2024-01-12T01:08:20.330 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55155.log 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46616.log.gz 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35370.log 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37383.log: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37383.log.gz 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62238.log 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55155.log.gz 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65560.log 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70704.log 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35370.log.gz 2024-01-12T01:08:20.331 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62238.log.gz 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41748.log 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65560.log.gz 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29157.log 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70704.log.gz 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53174.log /var/log/ceph/ceph-client.admin.41748.log: 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41748.log.gz 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29293.log 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29157.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.29157.log.gz 2024-01-12T01:08:20.332 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49837.log 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53174.log.gz 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43322.log 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29293.log.gz 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49837.log: /var/log/ceph/ceph-client.admin.34979.log 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49837.log.gz 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67317.log 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43322.log: /var/log/ceph/ceph-client.admin.34979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43322.log.gz 2024-01-12T01:08:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27594.log 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34979.log.gz 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56927.log 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67317.log.gz 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60702.log 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27594.log.gz 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45075.log 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56927.log.gz 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60702.log: /var/log/ceph/ceph-client.admin.24809.log 2024-01-12T01:08:20.334 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60702.log.gz 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66071.log 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45075.log.gz 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66909.log 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24809.log.gz 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37530.log 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66071.log.gz 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42763.log 2024-01-12T01:08:20.335 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66909.log.gz 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50030.log 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62603.log 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37530.log.gz 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42763.log.gz 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63312.log 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50030.log.gz 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67188.log 2024-01-12T01:08:20.336 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62603.log.gz 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52429.log 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63312.log.gz 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46082.log 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67188.log.gz 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48263.log 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52429.log.gz 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26672.log 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46082.log: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.46082.log.gz 2024-01-12T01:08:20.337 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33348.log 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48263.log.gz 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.26672.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53236.log 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26672.log.gz 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39842.log 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33348.log.gz 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40204.log 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53236.log.gz 2024-01-12T01:08:20.338 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52472.log 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39842.log.gz 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60059.log 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40204.log.gz 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65776.log 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52472.log: /var/log/ceph/ceph-client.admin.60059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31575.log 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52472.log.gz 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.60059.log.gz 2024-01-12T01:08:20.339 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65776.log.gz 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38152.log 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31575.log.gz 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27486.log 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38152.log.gz 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.44013.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63519.log 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2024-01-12T01:08:20.340 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63210.log 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27486.log.gz 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44702.log 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63519.log.gz 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63210.log.gz 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34333.log 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25611.log 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44702.log.gz 2024-01-12T01:08:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61900.log 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34333.log.gz 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68238.log 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25611.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61919.log 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25611.log.gz 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61638.log 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61900.log.gz 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68238.log.gz 2024-01-12T01:08:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37005.log 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61919.log.gz 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61638.log.gz 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28796.log 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37005.log: gzip /var/log/ceph/ceph-client.admin.40344.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33874.log 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37005.log.gz 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-01-12T01:08:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36690.log 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr: 3.8% -- replaced with /var/log/ceph/ceph-client.admin.28796.log.gz 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56648.log 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33874.log.gz 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67511.log 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36690.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.36690.log.gz 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38345.log 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56648.log.gz 2024-01-12T01:08:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48770.log 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67511.log.gz 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45011.log 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38345.log.gz 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50202.log 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48770.log.gz 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42870.log 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45011.log.gz 2024-01-12T01:08:20.345 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30215.log 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50202.log.gz 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35047.log 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42870.log.gz 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30079.log 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30215.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30215.log.gz 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27765.log 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35047.log.gz 2024-01-12T01:08:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46742.log 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30079.log.gz 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27765.log.gz 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50438.log 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27851.log 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46742.log.gz 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30504.log 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50438.log.gz 2024-01-12T01:08:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.27851.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43992.log 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27851.log.gz 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34214.log 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30504.log.gz 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52365.log 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.43992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43992.log.gz 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49350.log 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34214.log.gz 2024-01-12T01:08:20.348 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62035.log 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52365.log.gz 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41513.log 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49350.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.49350.log.gz 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40045.log 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62035.log.gz 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69973.log 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.41513.log.gz 2024-01-12T01:08:20.349 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68157.log 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40045.log.gz 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69973.log.gz 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47278.log 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68157.log.gz 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61598.log 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2024-01-12T01:08:20.350 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59063.log 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47278.log.gz 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59527.log 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61598.log.gz 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33568.log 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59063.log.gz 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59527.log.gz 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32628.log 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52515.log 2024-01-12T01:08:20.351 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33568.log.gz 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32628.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60622.log 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32628.log.gz 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65577.log 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52515.log.gz 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37173.log 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60622.log.gz 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.65577.log: --verbose -- /var/log/ceph/ceph-client.admin.69838.log 2024-01-12T01:08:20.352 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65577.log.gz 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56568.log 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr: 23.3% -- replaced with /var/log/ceph/ceph-client.admin.37173.log.gz 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53195.log 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69838.log.gz 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45999.log 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56568.log.gz 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64830.log 2024-01-12T01:08:20.353 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53195.log.gz 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27206.log 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35849.log 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.64830.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64830.log.gz 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45999.log.gz 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27206.log.gz 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64636.log 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41711.log 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35849.log: 22.2%/var/log/ceph/ceph-client.admin.64636.log: -- replaced with /var/log/ceph/ceph-client.admin.35849.log.gz 2024-01-12T01:08:20.354 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64636.log.gz 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26736.log 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59119.log 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41711.log.gz 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66587.log 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26736.log.gz 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25067.log 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59119.log.gz 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70379.log 2024-01-12T01:08:20.355 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66587.log.gz 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38367.log 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25067.log.gz 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33428.log 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70379.log.gz 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68783.log 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.38367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38367.log.gz 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29229.log 2024-01-12T01:08:20.356 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33428.log.gz 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57658.log 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68783.log.gz 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29229.log: /var/log/ceph/ceph-client.admin.66693.log 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29229.log.gz 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54811.log 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57658.log.gz 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.66693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39863.log 2024-01-12T01:08:20.357 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66693.log.gz 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54811.log.gz 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28848.log 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39863.log.gz 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68375.log 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49622.log 2024-01-12T01:08:20.358 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28848.log.gz 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24917.log 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68375.log.gz 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54218.log 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49622.log.gz 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58174.log 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24917.log.gz 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46353.log 2024-01-12T01:08:20.359 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54383.log 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54218.log.gz 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58174.log.gz 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28259.log 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46353.log.gz 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54383.log: /var/log/ceph/ceph-client.admin.48117.log 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54383.log.gz 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59983.log 2024-01-12T01:08:20.360 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28259.log.gz 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56181.log 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48117.log.gz 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.59983.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24353.log 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59983.log.gz 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25743.log 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56181.log.gz 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.24353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2024-01-12T01:08:20.361 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24353.log.gz 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25743.log: /var/log/ceph/ceph-client.admin.39036.log 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25743.log.gz 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29831.log 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37593.log: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gz 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32648.log 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55413.log 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-01-12T01:08:20.362 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31456.log 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32648.log: -- replaced with /var/log/ceph/ceph-client.admin.29831.log.gz 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32648.log.gz 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.55413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25298.log 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55413.log.gz 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58651.log 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31456.log: 2.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.31456.log.gz 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.51568.log 2024-01-12T01:08:20.363 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25298.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.25298.log.gz 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.58651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54261.log 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58651.log.gz 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25781.log 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51568.log.gz 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29809.log 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54261.log.gz 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65413.log 2024-01-12T01:08:20.364 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.25781.log.gz 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47127.log 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29809.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31439.log 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29809.log.gz 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65677.log 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47127.log: 11.1% -- replaced with /var/log/ceph/ceph-client.admin.65413.log.gz 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47127.log.gz 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54660.log 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.31439.log.gz 2024-01-12T01:08:20.365 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 /var/log/ceph/ceph-client.admin.65677.log: --verbose -- /var/log/ceph/ceph-client.admin.61364.log 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65677.log.gz 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30589.log 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54660.log: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.54660.log.gz 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30147.log 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61364.log.gz 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32668.log 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30589.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30589.log.gz 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53026.log 2024-01-12T01:08:20.366 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30147.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30147.log.gz 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54089.log 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49987.log 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32668.log.gz 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53026.log.gz 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44492.log 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54089.log.gz 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48410.log 2024-01-12T01:08:20.367 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.49987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49987.log.gz 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29680.log 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44492.log.gz 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66265.log 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26844.log 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48410.log.gz 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29680.log.gz 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61012.log 2024-01-12T01:08:20.368 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70977.log /var/log/ceph/ceph-client.admin.66265.log: 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66265.log.gz 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26844.log.gz 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61012.log.gz 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27379.log 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45918.log 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70977.log.gz 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30334.log 2024-01-12T01:08:20.369 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27379.log.gz 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37404.log 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45918.log.gz 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65045.log 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30334.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30334.log.gz 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35723.log 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37404.log: 23.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49179.log 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37404.log.gz 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26586.log 2024-01-12T01:08:20.370 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65045.log.gz 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35723.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.35723.log.gz 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49179.log: /var/log/ceph/ceph-client.admin.69676.log 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49179.log.gz 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44208.log 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26586.log.gz 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.69676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33148.log 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69676.log.gz 2024-01-12T01:08:20.371 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.44208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44208.log.gz 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51238.log 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33148.log.gz 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42229.log 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51238.log.gz 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33925.log 2024-01-12T01:08:20.372 INFO:teuthology.orchestra.run.smithi178.stderr: 31.5% -- replaced with /var/log/ceph/ceph-client.admin.42229.log.gz 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51934.log 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54721.log 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33925.log.gz 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48942.log 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.51934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51934.log.gz 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25763.log 2024-01-12T01:08:20.373 INFO:teuthology.orchestra.run.smithi178.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.54721.log.gz 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23757.log 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.48942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48942.log.gz 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53404.log 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25763.log.gz 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23757.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.31592.log -- replaced with /var/log/ceph/ceph-client.admin.23757.log.gz 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.53404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30181.log 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53404.log.gz 2024-01-12T01:08:20.374 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42249.log 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31592.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31592.log.gz 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30181.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30181.log.gz 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27637.log 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26008.log 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42249.log.gz 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36984.log 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.27637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27637.log.gz 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.26008.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65372.log 2024-01-12T01:08:20.375 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26008.log.gz 2024-01-12T01:08:20.376 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50498.log 2024-01-12T01:08:20.376 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.36984.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.36984.log.gz 2024-01-12T01:08:20.376 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32688.log 2024-01-12T01:08:20.376 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.65372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65372.log.gz 2024-01-12T01:08:20.376 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62091.log 2024-01-12T01:08:20.376 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50498.log: 0.0%/var/log/ceph/ceph-client.admin.32688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28324.log 2024-01-12T01:08:20.376 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50498.log.gz 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32688.log.gz 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29508.log 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.62091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62091.log.gz 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67360.log 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28324.log.gz 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30827.log 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29508.log.gz 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2024-01-12T01:08:20.377 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.67360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67360.log.gz 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23830.log 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30827.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.30827.log.gz 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47299.log 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61558.log 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.23830.log: /var/log/ceph/ceph-client.admin.47299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23830.log.gz 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31868.log 2024-01-12T01:08:20.378 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47299.log.gz 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61558.log.gz 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33228.log 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31868.log.gz 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70163.log 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.39987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60505.log 2024-01-12T01:08:20.379 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33228.log.gz 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.70163.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39079.log 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70163.log.gz 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47041.log 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60505.log: 0.0%/var/log/ceph/ceph-client.admin.39079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46536.log 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60505.log.gz 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39079.log.gz 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61174.log 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.47041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47041.log.gz 2024-01-12T01:08:20.380 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68726.log 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46536.log.gz 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57035.log 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61174.log.gz 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70271.log 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.68726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68726.log.gz 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61521.log 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57035.log.gz 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70271.log: /var/log/ceph/ceph-client.admin.32048.log 2024-01-12T01:08:20.381 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70271.log.gz 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26607.log 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.61521.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63776.log 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32048.log: -- replaced with /var/log/ceph/ceph-client.admin.61521.log.gz 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32048.log.gz 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63647.log 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26607.log.gz 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31269.log 2024-01-12T01:08:20.382 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63776.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60680.log 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63776.log.gz 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.63647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70650.log 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.31269.log: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.63647.log.gz 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.31269.log.gz 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54046.log 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53960.log 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr: 55.4% -- replaced with /var/log/ceph/ceph-client.admin.60680.log.gz 2024-01-12T01:08:20.383 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.70650.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70650.log.gz 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.54046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54046.log.gz 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27443.log /var/log/ceph/ceph-client.admin.53960.log: 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53960.log.gz 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41670.log 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.27443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28967.log 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27443.log.gz 2024-01-12T01:08:20.384 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46229.log 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.41670.log: 31.8% -- replaced with /var/log/ceph/ceph-client.admin.41670.log.gz 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.28967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28967.log.gz 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46229.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59101.log 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45505.log 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.40667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-01-12T01:08:20.385 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.59101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60362.log 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59101.log.gz 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.45505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45505.log.gz 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33328.log 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.60362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60362.log.gz 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32488.log 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42349.log 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33328.log.gz 2024-01-12T01:08:20.386 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.32488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33768.log 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32488.log.gz 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.42349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52773.log 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42349.log.gz 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.33768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33768.log.gz 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35618.log 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.52773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30521.log 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52773.log.gzgzip 2024-01-12T01:08:20.387 INFO:teuthology.orchestra.run.smithi178.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.50358.log 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35618.log: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.35618.log.gz 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30521.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30521.log.gz 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.50358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50358.log.gz 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57723.log 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr:gzip /var/log/ceph/ceph-client.admin.38928.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29210.log 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-01-12T01:08:20.388 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56424.log 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.57723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57723.log.gz 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46679.log 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.29210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29210.log.gz 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56424.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37894.log 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56424.log.gz 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.46679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46679.log.gz 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56777.log 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.37894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26199.log 2024-01-12T01:08:20.389 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2024-01-12T01:08:20.390 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37894.log.gz 2024-01-12T01:08:20.390 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.56777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56777.log.gz 2024-01-12T01:08:20.390 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.26199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26199.log.gz 2024-01-12T01:08:20.390 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.30844.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2024-01-12T01:08:20.900 INFO:teuthology.orchestra.run.smithi178.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-12T01:08:22.680 INFO:teuthology.orchestra.run.smithi178.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-12T01:08:23.797 INFO:teuthology.orchestra.run.smithi178.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-12T01:08:46.248 INFO:teuthology.orchestra.run.smithi178.stderr: 96.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-12T01:08:58.456 INFO:teuthology.orchestra.run.smithi178.stderr: 96.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-12T01:08:58.456 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-12T01:08:58.456 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m39.328s 2024-01-12T01:08:58.457 INFO:teuthology.orchestra.run.smithi178.stderr:user 1m12.891s 2024-01-12T01:08:58.457 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m3.498s 2024-01-12T01:08:58.458 INFO:tasks.ceph:Archiving logs... 2024-01-12T01:08:58.458 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820/remote/smithi178/log 2024-01-12T01:08:58.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-12T01:09:08.203 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2024-01-12T01:09:08.205 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-12T01:09:08.225 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2024-01-12T01:09:08.227 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-12T01:09:08.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-12T01:09:08.271 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-12T01:09:08.280 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-12T01:09:08.281 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-12T01:09:08.304 INFO:teuthology.orchestra.run.smithi178.stdout: remote refid st t when poll reach delay offset jitter 2024-01-12T01:09:08.305 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================== 2024-01-12T01:09:08.305 INFO:teuthology.orchestra.run.smithi178.stdout:*hv01.front.sepi 44.4.53.2 2 u 503 512 377 0.168 0.647 0.137 2024-01-12T01:09:08.305 INFO:teuthology.orchestra.run.smithi178.stdout:+hv02.front.sepi 63.231.80.2 3 u 451 512 377 0.175 0.351 0.048 2024-01-12T01:09:08.305 INFO:teuthology.orchestra.run.smithi178.stdout:+hv03.front.sepi 74.6.168.72 3 u 468 512 377 0.149 -0.351 0.226 2024-01-12T01:09:08.305 INFO:teuthology.orchestra.run.smithi178.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-01-12T01:09:08.306 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-12T01:09:08.325 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-12T01:09:08.325 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-12T01:09:08.335 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-12T01:09:08.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-12T01:09:08.353 INFO:teuthology.task.internal:Duration was 12758.307073 seconds 2024-01-12T01:09:08.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-12T01:09:08.362 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-12T01:09:08.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-12T01:09:08.415 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-12T01:09:08.415 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-01-12T01:09:08.416 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-12T01:09:08.427 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-12T01:09:08.427 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-12T01:09:08.456 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-12T01:09:08.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-12T01:09:08.522 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-12T01:09:08.533 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-12T01:09:08.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-12T01:09:08.550 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-12T01:09:08.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-12T01:09:08.605 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-01-12T01:09:08.630 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-12T01:09:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-12T01:09:08.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-12T01:09:08.647 INFO:teuthology.task.internal:Transferring archived files... 2024-01-12T01:09:08.648 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820/remote/smithi178 2024-01-12T01:09:08.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-12T01:09:08.674 INFO:teuthology.task.internal:Removing archive directory... 2024-01-12T01:09:08.674 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-12T01:09:08.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-12T01:09:08.728 INFO:teuthology.task.internal:Not uploading archives. 2024-01-12T01:09:08.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-12T01:09:08.738 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-12T01:09:08.738 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-12T01:09:08.746 INFO:teuthology.orchestra.run.smithi178.stdout: 924112 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 12 01:09 /home/ubuntu/cephtest 2024-01-12T01:09:08.746 INFO:teuthology.orchestra.run.smithi178.stdout: 1069127 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 11 21:44 /home/ubuntu/cephtest/mnt.0 2024-01-12T01:09:08.746 INFO:teuthology.orchestra.run.smithi178.stdout: 1069128 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 11 21:45 /home/ubuntu/cephtest/mnt.0/client.0 2024-01-12T01:09:08.746 INFO:teuthology.orchestra.run.smithi178.stdout: 1195570 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 11 21:45 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-12T01:09:08.747 INFO:teuthology.orchestra.run.smithi178.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-01-12T01:09:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-12T01:09:08.748 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_cbodley_ceph_49908e15c7044e78475683aa6ccd77bd187be2f4/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-01-12T01:09:08.761 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-12T01:09:08.771 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-12T01:09:08.781 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2024-01-12T01:09:08.782 INFO:teuthology.nuke:Checking targets against current locks 2024-01-12T01:09:08.811 DEBUG:teuthology.nuke:shortname: smithi178 2024-01-12T01:09:08.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-12T01:09:08.834 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-11_21:24:20-rbd-quincy-distro-default-smithi/7513820', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 21:26:49.077035', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2024-01-12T01:09:08.883 INFO:teuthology.orchestra.console.smithi178:Power off 2024-01-12T01:09:08.883 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-12T01:09:08.907 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-01-12T01:09:08.908 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-12T01:09:08.931 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-01-12T01:09:12.932 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-12T01:09:13.043 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-01-12T01:09:17.045 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-12T01:09:17.156 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-01-12T01:09:17.257 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-01-12T01:09:17.363 INFO:teuthology.run:Summary data: description: rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic} duration: 12758.307073354721 failure_reason: 'Command failed (workunit test rbd/cli_generic.sh) on smithi178 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=49908e15c7044e78475683aa6ccd77bd187be2f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=77c422986b824c2b950f161693dd0696 status: fail success: false 2024-01-12T01:09:17.363 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-12T01:09:17.585 INFO:teuthology.run:FAIL